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PREFACE 


The  work  described  in  this  report  was  authorized  under  Army  Science  and 
Technology  Objective  (STO)  IV.ME.2004.03.  The  woric  described  in  this  report  was  started  in 
January  2004  and  completed  in  September  2005. 

This  final  report  was  prepared  in  response  to  a  request  fijom  the  U.S.  Army  Center 
for  Environmental  Health  Research  to  develop  a  methodology  to  evaluate  Environmental 
Sentinel  Biomonitor  (ESB)  technologies  and  then  to  evaluate  and  downselect  the  most 
appropriate  ESB  technologies  to  further  develop  into  an  ESB  system. 

The  use  of  either  trade  or  manufacturers’  names  in  this  report  does  not  constitute 
an  official  endorsement  of  any  commercial  products.  This  report  may  not  be  cited  for  purposes 
of  advertisement. 

This  report  has  been  approved  for  public  release.  Registered  users  should  request 
additional  copies  of  this  report  fi-om  the  Defense  Technical  Information  Center;  unregistered 
users  should  direct  such  requests  to  the  National  Technical  Information  Service 
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ENVIRONMENTAL  SENTINEL  BIOMONTTOR  (ESB)  SYSTEM 
TECHNOLOGY  ASSESSMENT 


1.  OVERVIEW 

The  Environmental  Sentinel  Biomonitor  (ESB)  system  downselection  process  was 
conducted  in  FY04  and  FY05.  An  initial  technical  report  described  work  completed  in  FY04 
and  the  plan  to  complete  tiie  project  in  FY05.*  Information  in  this  report  takes  precedence  over 
the  initial  technical  report  and  describes  the  methodology  and  results  for  the  entire  project’s 
effort. 

1.1  Introduction. 

The  U.S.  Army  Center  for  Environmental  Health  Research  (USACEHR),  with 
support  from  Army  chent  organizations  and  funding  from  Army  Technology  Objective  (ATO) 
IV.MD.2004.03,  is  developing  an  ESB  system  to  provide  rapid  toxicity  identification  for  a  broad 
spectrum  of  chemicals  in  water.  The  focus  of  the  ESB  system  is  to  detect  toxicity  associated 
with  non-militarized  chemicals  (i.e.,  toxic  industrial  chemicals  [TICs]  and  toxic  industrial 
materials  [TIMs])  in  water. 

A  critical  initial  phase  of  the  ATO  is  to  test  and  evaluate  ESB  technologies. 
Because  there  are  a  munber  of  potentially  feasible  technologies  that  could  meet  the  goals  of  the 
ESB  program,  a  downselect  was  performed  to  evaluate  these  technologies  and  select  the  most 
promising  ones  for  further  development  as  part  of  an  ESB  system. 

1.2  Background. 

Deployed  U.S.  forces  face  the  possibility  of  drinking  water  exposed  to  a  wide 
range  of  toxic  industrial  or  agricultural  chemicals  as  a  result  of  normal  use  (e.g.,  farm  run-off), 
damaged  infrastructures,  accidental  ^ills,  or  deliberate  chemical  contamination  of  water. 
Unfortunately,  rapid  detection  c^abilities  for  toxic  chemicals  in  water  are  limited  and  may  not 
provide  sufficient  warning  of  developing  toxic  hazards. 

The  ESB  system  is  intended  to  monitor  responses  of  biological  components  (e.g., 
enzymes,  cells,  tissues,  or  whole  organisms)  exposed  to  water  and  to  provide  rapid 
rei^nses/wamings  should  toxic  conditions  develop.  Cell-based  sensors  are  becoming  available 
that  integrate  biological  systems  with  electronic  monitoring,  facilitating  rapid  response  to 
developing  toxicity  in  water  using  very  small  systems. 


*  Kooistra,  S.;  Walther,  J.;  Initial  Technology  Assessment  For  The  Environmental  Sentinel 
Biomonitor  (ESB)  System;  ECBC-TR-477;  U.S.  Army  Edgewood  Chemical  Biological  Center: 
Aberdeen  Proving  Ground,  MD,  2006;  UNCLASSIFIED  Report  (AD-A454  235). 
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1.3 


ESB  Program  Objective. 


The  ESB  program  will  incorporate  current  toxicity  sensor  technologies  into  an 
ESB  system  having  size,  weight,  and  logistical  characteristics  suitable  for  a  range  of  Army 
requirements  that  will  complement  current  chemical  monitoring  systems  and  provide  rapid 
toxicity  identification  for  a  broad  spectrum  of  chemicals  in  water.  The  optimal  system  may  be  a 
complementary  set  of  toxicity  sensors,  which  would  provide: 

•  Rapid  response.  Required  response  times  may  range  fi-om  a  few  minutes  to  an 
hour  depending  upon  the  particular  Army  use  scenario. 

•  Sensitivity.  One  ESB  technology  may  not  adequately  detect  all  TICs/TIMs. 
The  assessment  will  consider  not  only  which  ESB  technology  provides  the 
best  overall  response  to  the  test  chemicals,  but  also  which  set  of  technologies 
can  complement  each  other  by  filling  gaps  in  toxicity  response  for  individual 
technologies  as  well  as  providing  mutual  confirmation  of  a  toxic  response. 

1.4  Assessment  Process  Overview\ 

A  decision  analysis-based  methodology  was  developed  to  conduct  the  ESB 
system  downselection.  Decision  analysis  is  a  structured  process  for  decision-making  based  on 
established  principles  of  operations  research.  The  decision  analysis  process  includes  systematic 
development  and  examination  of  alternative  courses  of  action  to  define  and  clarify  available 
choices  and  associated  advantages  and  disadvantages.  It  also  includes  thorough  documentation 
of  results  and  associated  rationale  so  that  final  recommendations  can  be  readily  explained  and 
defended.  The  study  consisted  of  six  phases  (described  in  sections  2. 1-2.6). 

1 .  Form  study/assessment  team. 

2.  Develop  ESB  system  applications  (i.e.,  scenarios). 

3.  Define  technical  requirements  of  ESB  system  for  each  application. 

4.  Identify  candidate  ESB  technologies.  Develop  and  use  a  screening  model  to 
reduce  number  of  technologies  to  consider.  Collect  missing  toxicity  response 
data. 

5.  Develop  quantitative  assessment  models. 

6.  Generate  and  analyze  results  using  quantitative  assessment  model. 


2.  ASSESSMENT  PROCESS  AND  RESULTS 

2.1  Studv/Assessment  Team  Formed. 

An  assessment  team  was  formed  in  early  FY04  to  conduct  the  downselect.  The 
team  was  led  by  Dr.  William  van  der  Schalie  (USACEHR)  and  Dr.  Thomas  Gargan  II 
(USACEHR),  with  support  fi'om  the  Decision  Analysis  Team  (DAT)  of  the  Edgewood  Chemical 
and  Biological  Center.  The  assessment  team  was  made  up  of  Army  user  representatives 
(members  of  an  Integrated  Product  Team)  and  technical  experts  fi^om  collateral  organizations  and 
academia  (see  Appendix  A  for  a  list  of  team  members  and  their  roles). 
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The  role  of  the  user  representatives  was  to  articulate  the  applications/scenarios  for 
all  possible  Anny  users  of  an  ESB  system.  They  also  defined  the  technical  requirements  for  the 
ESB  system  for  each  scenario.  Their  final  task  was  to  help  develop  the  quantitative 
downselection  assessment  model. 


The  role  of  the  technical  expert  was  to  be  knowledgeable  about  the  ESB 
technologies,  help  develop  the  assessment  models,  and  fiien  assess  the  technologies  against  the 
models.  Although  the  user  representatives  have  primary  responsibility  for  model  development, 
the  technical  expert’s  input  was  important  because  they  provided  input  on  the  technical 
feasibility  of  measures  for  the  models. 


2.2  ESB  System  Applications/Scenarios  Defined. 


Several  Army  applications  for  an  ESB  system  were  identified  early  in  the  ATO 
program  process.  These  include; 


•  Use  in  conjunction  with  the  Tactical  Water  Purification  System  (TWPS)  and 
Light  Weight  Water  Purification  System,  including  assessment  of  water  pre- 
and  post-treatment  and  supporting  decisions  regarding  the  suitability  of  using 
a  fieshwater  bypass  system.  Use  of  an  ESB  system  is  consistent  with  a 
recommendation  in  the  Pre-Planned  Product  hnprovement  section  of  the 
TWPS  Operational  Requirements  Document. 

•  Testing  water  produced  by  on-board  water  generation  equipment  in  Manned 
Groimd  Vehicles  being  developed  under  the  Future  Combat  Systems  program. 

•  Use  by  Preventive  Medicine  personnel  to  evaluate  potable  water  quality  in  a 
variety  of  situations. 

•  Use  at  Army  installations,  both  domestically  and  overseas,  to  evaluate  source 
and  drinking  water  quality. 


The  user  representatives  reviewed  the  above  Army  applications  and  identified 
others.  They  then  analyzed  and  groiq)ed  these  apphcations  to  form  the  fewest  number  of 
scenarios  that  would  represent  all  identified  Army  applications.  This  was  done  through  an 
iterative  process  and  resulted  in  four  basic  scenarios  to  represent  the  situations  in  which  the 
Army  operates.  Two  of  the  four  basic  scenarios  were  further  subdivided  by  type  of  water  tested 
and  collection  method  to  create  two  more  scenarios,  for  a  total  of  six. 


The  type  of  water  tested  coiild  either  be  raw  (i.e.,  water  that  has  not  been  treated 
with  any  mechanism  [e.g.,  filters]  or  chemicals  [e.g.,  chlorine])  or  treated  (e.g.,  water  treated 
with  a  TWPS,  hand-held  water  treataiCTit  device,  or  chemicals). 

There  are  two  types  of  collection  methods.  In  the  “continuous  flow”  method,  for 
example,  the  water  must  be  continually  tested  as  the  water  flows  inside  a  pipeline  from  one  point 
to  the  next.  An  ESB  system  using  the  continuous  flow  method  must  constantly  monitor  the 
water  for  contamination.  The  results  of  the  tests  need  to  be  available  in  near  real-time  to  prevent 
use  or  contamination  further  along  in  the  distribution  system.  The  “grab  sample”  method  is 
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defined  as  procuring  water  fi'om  a  source  and  then  testing  it  off-line.  The  water  needs  to  be 
collected  and  tested  before  it  is  used,  with  maximum  response  times  ranging  between  a  few 
minutes  to  an  hour. 

The  six  scenarios  are  presented  as  an  overview  in  Figure  1  and  further  described 
in  the  rest  of  section  2.2.1.  Figure  1  briefly  describes  each  scenario  and  notes  visually  with  gray 
boxes  the  sample  collect  methods  and  type  of  water  the  ESB  technology  would  test.  For 
example,  for  the  Individual/Small  Team  scenario  the  sample  collection  method  is  a  grab  sample 
and  the  type  of  water  tested  is  both  raw  and  treated. 


Sample  Collection 
Method 

Type  of  Water  Tested 

ESSl 

Scenario  Overview 

Grab 

Contin¬ 

uous 

Raw’ 

Water 

Treated 

Water 

ESB  system  would  be  transported  and 
operated  by  each  individual. 

Small  Unit 

ESB  system  would  be  transported  by  a 
vehicle,  which  has  limited  capacity  to  carry 
an  ESB  system.  ESB  system  would  be 
operated  by  anyone  within  team. 

Field/ 

Contingency 
(grab  sample, 
raw  or  treated 
water) 

ESB  system  would  be  transported  by 
vehicle  (e.g.,  truck,  C-130)  with  significant 
more  capacity  to  carry  an  ESB  system  than 
in  the  Individual/Small  Team  and  Small 
Unit  scenarios.  The  ESB  system  can  be 
larger  and  heavier.  ESB  system  would  be 
operated  by  personnel  of  higher  grades 
and/or  skills  so  a  more  complicated  ESB 
system  than  in  die  Individual/Small  Team 
and  Small  Unit  scenario  could  be 
operated/used. 

Field/ 

Contingency 
(continuous 
sample,  treated 
water) 

1 . . 

1  1  iff J.Til  iFJI  FiltU  111  ■  i  tM 

Figure  1:  OvmdewofSix  Scenarios  Developed 


2.1 


Individual/Small  Team  Scenario. 


This  scenario  represents  one  Soldier  or  a  small  team  of  Soldiers  operating 
motely  away  from  a  fixed  or  field/contingency  base.  In  this  scenario  each  Soldier  can  operate 
dependently  because  they  have  their  own  ESB  system.  An  example  is  a  small  team  of  Soldiers 
)erating  remotely  in  the  hills  of  Afghanistan  -  Soldiers  are  flown  in  and  out  of  this  remote 
cation  every  14  days;  they  must  procure  local  water  each  day. 


The  ESB  system  provides  the  individual  Soldier  or  small  temn  the  ability  to  test  a 
ab  sample  of  raw  or  treated  water  in  the  field  for  flie  presence  of  toxic  chemicals  that  could 
use  acute  health  effects  and  degrade  battlefield  readiness/performance.  The  Soldier  could  test 
w  water  fix)m  a  surface  water  source  prior  to  disinfecting  and  drinking  it  or  test  water  that  has 
«n  treated  using  a  hand-held  water  treatment  device  (e.g.,  Army’s  Li^t  Wei^t  Water 
irification  System).  This  gives  the  individual  Soldier  an  enhanced  capability  to  check  the 
lality  of  his  personally  acquired/produced  water  before  consuming  it.  The  ESB  system  would 
:  used  in  emergency  situations  when  the  Soldier  does  not  have  access  to  a  doctrinally  treated 
id/or  distributed  water  supply  and  must  acquire,  produce,  or  purify  his/her  own  water  (e.g., 
itial  entry,  remote  site,  and  Special  Operations  Forces  missions). 

2.2  Small  Unit  Scenario. 

This  scenario  is  similar  to  the  individual/small  team  scenario.  Instead  of  an 
dividual  operating  with  only  what  they  can  carry  or  procure  locally  (e.g.,  local  untreated 
Iter),  the  Soldiers  are  operating  within  a  small  unit  construct.  The  ESB  system  would  be 
insported  within  a  vehicle.  Water  could  be  procured  the  same  way  as  with  the 
dividual/Small  Team  scenario  or  through  water  produced  from  the  Future  Combat  Systems  - 
anned  Ground  Vehicle  (MGV).  The  MGV  has  flie  ability  to  generate  potable  water  throu^  a 
acess  that  starts  with  capturing  the  exhaust  of  the  vehicle.  The  purpose  of  the  ESB  system  in 
IS  situation  would  be  to  independently  verify  that  the  water  produced  by  the  MGV  is  potable. 

2.3  Field/Contingencv  Scenarios. 

There  are  two  scenarios,  differing  only  in  the  type  of  water  tested  and  collection 
jthod,  which  represent  a  field/contingency  operation.  In  the  Field/Contingency  scenarios 
)re  than  a  small  team  or  unit  of  Soldiers  is  living  and  working,  for  more  than  1 5  days,  and  their 
iter  is  procured  from  a  questionable  source  (quality/safety  of  water  is  not  known).  The  first 
mario  collects  water  with  a  grab  sample  and  tests  both  raw  and  treated  water.  An  example  of 
5  first  scenario  is  a  reconnaissance  team,  in  its  mission  to  find  a  suitable  source  of  water  for  the 
verse  Osmosis  Water  Purification  Unit  (ROWPU)  or  TWPS  units,  must  test  existing  water 
irces  in  the  area  of  the  field/contingency  base.  Another  example  is  Preventive  Medicine 
rsonnel  need  to  periodically  check  for  TICs/TIMs  in  storage  tanks  and  distribution  systems, 
e  objective  is  to  rapidly  identify  accidental  or  intentional  spills,  discharges  or  contamination. 

The  second  scenario  collects  water  with  a  continuous  flow  sample  and  tests  only 
ated  water.  An  example  of  the  second  scenario  is  quartermaster  personnel  using  a  continuous 
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flow  ESB  system  to  monitor  both  water  that  is  the  source  for  their  ROWPU  or  TWPS  as  well  as 
the  water  produced  by  these  units/systems. 

2.2.4  Fixed  Base  Scenarios. 

There  are  two  scenarios,  differing  only  by  type  of  water  tested  and  collection 
method,  which  represent  operations  at  a  permanent  base  either  within  the  United  States  or 
overseas  or  at  rear  areas  in  deployed  situations.  In  these  scenarios,  the  Soldiers  may  obtain  their 
drinking  water  from  water  treatment  plants  that  meet  U.S.  quality  standards  (potable  water  is  not 
being  created  with  a  ROWPU  or  TWPS).  The  water  treatment  plant  could  be  located  either  on 
base  or  off  base,  with  the  water  piped  in.  The  population  will  be  less  homogeneous  than  the 
other  scenarios  if  dependents  are  using  the  water  supply.  An  example  of  the  first  scenario,  grab 
Sample  to  Test  Raw  or  Treated  Water,  is  Public  Woiks,  Preventive  Medicine,  and  possibly 
Security  personnel  who  must  perform  spot  checks  of  water  system  assets  (e.g.,  treatment  plant 
product  water,  individual  storage  tanks,  distribution  system)  for  the  presence  of  TICs/TIMs  at 
levels  that  could  cause  harmful  health  effects.  An  example  of  the  second  scenario,  continuous 
flow  sample  to  test  raw  or  treated  water,  is  installation  Public  Works  personnel  who  must 
monitor  continuously  for  the  introduction  of  TICs/TIMs  into  the  water  supply  system  in  order  to 
prevent  or  minimize  problems. 

2.3  Technical  Requirements  Defined. 

The  user  representatives  developed  technical  requirements  for  each  of  the  six 
scenarios.  These  requirements  define  in  detail  the  performance  specifications  for  the  ESB 
system  and  are  made  up  of  26  questions  (covering  eight  assessment  categories)  and  their  answers 
(see  Appendix  B).  The  piupose  of  this  document  is  to  provide  more  specific  information  for 
each  of  the  six  scenarios  and  to  help  the  technical  experts  know  what  information  must  be 
collected  (throu^  research  or  additional  testing)  for  the  downselection  evaluation.  The 
document  is  not  intended  to  be  the  final  design  specification  for  an  ESB  system.  The  user 
representatives  noted  both  threshold/minimum  requirements  and  ideal/preferred  requirements. 

Selected  and  Defined  Target  Detection  Ranee. 

The  concentration  range  over  which  ESB  technologies  must  detect  TICs/TIMs 
(upper  and  lower  limit)  needed  to  be  specified  before  the  technologies  could  be  evaluated.  The 
user  representatives  agreed  the  threshold  requirement  for  the  ESB  system  technologies  is  to 
detect  TICs/TIMs  between  the  short-term  Military  Exposure  Guidelines  (MEGs)  level  (standard 
is  based  on  a  70  kg  Soldier  drinking  15  liters/day  for  7-14  days;  see  Appendix  C  for  additional 
information  about  MEGs)  and  the  human  lethal  concentration  (HLC),  which  assumes 
consumption  of  15  liters  in  one  day  by  a  70  kg  Soldier.  The  ESB  program  will  not  focus  on 
detecting  TICs/TIMs  at  more  sensitive  levels  (e.g.,  long-term  MEG  or  U.S.  Environmental 
Protection  Agency  measures  where  the  focus  is  on  a  chronic  effects  level)  imtil  more  promising 
ESB  technologies  become  available.  Until  then,  existing  analytical  chemistry  tools  will  be  used 
to  detect  TICs/TIMs  at  chronic  effect  levels. 


The  user  representatives  noted  that  ESB  technologies  should  detect  toxicity  closer 
to  the  short-term  MEG  level  than  the  HLC  level  and  that  the  minimum  detection  level  must  be 
below  the  HLC.  Finally,  user  representatives  stated  that  detecting  toxicity  below  the  short-term 
MEG  level  is  not  desirable  because  this  would  resuU  in  false  positive  readings. 

2.4  Candidate  ESB  Technologies  Identified.  Technologies  Eliminated  with  Screening 

Model,  and  Missing  Data  Collected. 

After  extensive  research,  40  possible  ESB  system  technologies  were  identified 
(listed  in  Appendix  D)  for  further  research  and  evaluation.  Fact  sheets  were  created  for  each 
technology  (see  Appendix  E  for  an  example)  and  presented  to  the  technical  experts.  The 
information  within  the  fact  sheets  encompassed  performance,  operational,  and  logistical 
characteristics. 


Through  the  research  process  of  developing  the  ESB  technology  fact  sheets  it  was 
determined  there  was  insufficient  toxicity  response  information  for  a  thorou^  comparative 
assessment  of  the  ESB  technologies.  To  complete  the  final  ESB  technology  assessment,  missing 
toxicity  response  data  had  to  be  collected  in  order  to  complete  the  final  technology  evaluation. 
Funding  and  time  constraints  did  not  allow  for  the  collection  of  toxicity  response  data  for  all  40 
technologies. 

2.4.1  Reduced  ESB  Technologies  with  a  Qualitative  Screening  Model. 

A  qualitative  screening  assessment  model  was  developed  and  used  to  reduce  the 
number  of  ESB  technologies  to  consider  further.  ESB  technologies  were  eliminated  from  further 
consideration  if  the  technical  experts  were  confident  the  technology  could  not  help  meet  the 
ATO. 

The  screening  model  iised  a  two-step  process.  In  the  first  step,  ESB  technologies 
were  eliminated  because  of  any  reason  listed  below. 

•  Technology  toxicity  sensitivity  is  inappropriate  or  the  technology  does  not 
provide  a  unique  end-point  (i.e.,  metal^lic,  physical  response). 

•  Technology  is  redundant  and  inferior  to  another  technology. 

•  Technology  is  unfeasible/not  viable.  It  does  not  or  is  not  expected  to  meet  a 
minimum  requirement  of  a  user  scenario. 

-Technology  takes  too  much  time  to  produce  an  end-point. 

-Technology  requires  pre-culturing  (e.g.,  requires  actively  taking  out  and 
culturing  organism  for  a  long  period  of  time). 

Twenty-three  of  the  40  technologies  were  eliminated  by  this  first  step.  In  the  final 
step,  the  17  remaining  technologies  were  evaluated  against  four  high  level  measures,  and  a  final 
ranking  was  generated.  The  four  high  level  measures  are: 

•  Technology  provides  an  appropriate  toxicity  sensitivity  response  (SR)  and/or 
provides  unique  information  (e.g.,  unique  end-point). 
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•  Technology  is  only  minimally  affected  by  interferences  (I).  Interferences  are 
turbidity,  color,  and  ionic  composition  of  the  media. 

•  Technology  is  reliable/results  are  repeatable  (R&R). 

•  Technology  has  an  appropriate  rapidity  of  response  (RoR).  A  green  rating  is  20 
minutes  or  less,  yellow  is  20-60  minutes,  and  red  is  over  60  minutes. 


Figure  2  provides  an  example  of  the  results  from  using  the  qualitative  screening 
assessment  model.  See  Appendix  F  for  details  on  definitions  of  assessment  measures  and  the 
assessment  methodology  and  Appendix  G  for  the  results  of  the  screening  process. 


SCREENEVG  ASSESSMENT/RESULTS  FOR  ESB  SYSTEM  TECHNOLOGIES 

Technology 

Biological 

System 

Step  1 
Rating 
(see 
Note  1) 

Step  1 

Rationale  and 
Comments 

Step  2  Rating  (see  Note  1  & 
Note  2) 

Step  2 

Rationale  and 
Comments 

SR 

B 

R&R 

RoR 

Mitoscan 

Enzyme  (sub- 
mito- 
chondrial 
particles) 

Green- 
Go  to 
Step  2 

Mammalian 
system  may 
provide  some 
unique 
sensitivities. 

1 

1 

Yellow 

1 

1 

1 

SR-  one  of  the  best 
technologies  under 
review;  I  -  expected 
to  be  similar  to 
Aquanox,  but  no 
data  available; 

R&R-  thought  to  be 
high,  SMPs;  RoR- 
range  of  5-20 
minutes 

Note  1:  See  Appendix  F  for  a  detailed  description  of  assessment  measure  definitions  and  assessment 
methodology  for  both  Steps  1  and  2. 

Note  2:  The  list  of  Step  2  screening  measures  were: 

1)  sensitivity  response  and/or  provides  unique  information  [SR], 

2)  handling  of  interferences  [I], 

3)  reliable/repeatable  [R&R],  and 

4)  rapidity  of  response  [RoR]. 

Figure  2:  ScreeningAssessment  Results  Example 


promising  (1 


Figure  3  lists  the  seventeen  ESB  technologies  that  were  determined  to  be  the  most 
is  most  promising,  17  least  promising)  by  the  technical  experts. 
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Rank 

Technology 

1 

Eclox 

2 

Aquanox 

3 

Microtox 

4 

Mitoscan  Electron  Transfer  (ETR) 

5 

Mitoscan  Reverse  Electron  Transfer  (RET) 

6 

ToxScreen  Metal 

7 

ToxScreen  Organic 

8 

Sinorhizobium  meliloti  Assay 

9 

SOS  Cytosensor 

10 

Portable  Neuronal  Microelectrode  Array 

11 

Portable  Cell-based  Biosensor 

12 

Cellsense 

13 

Amtox 

14 

Toxi-chromo  Test 

15 

Fluotox 

16 

ArrayScan  HCS  System 

17 

Electric  Cell-Substrate  Impedance  Sensing 
using  Endothelial  Cells 

F^;uiie3:  ESBTedmolo^esClwsen  with  Qualitative  ScreeiingModd 
For  the  reasons  stated  in  Figure  4,  six  technologies  were  dropped  from  further 

consideration. 


Technology 

Reason  Technology  was  Eliminated 

Aquanox 

Technical  experts  determined  Aquanox  and  Eclox  are  almost 
the  same  in  the  test  system  used  and  performance  and  decided 
only  one  of  the  technologies  should  be  further  studied.  With 
additional  evaluation,  Eclox  was  determined  to  be  sli^tly 
better,  so  Aquanox  was  dropped  from  further  consideration. 

Portable  Cell-  based 
Biosensor 

The  owner  of  technology  was  unable  to  participate  in  further 
testing. 

Cellsense 

European-based  technology;  the  owner  of  this  technology  was 
not  responsive  to  a  request  to  participate  in  further  testing. 

Amtox 

Extremely  costly  to  test;  not  a  hi^  enough  priority  but  may 
consider  testing  later  if  funding  becomes  available. 

Fluotox 

European-based  technology,  the  owner  of  this  technology  was 
not  responsive  to  a  request  to  participate  in  further  testing. 

Array  Scan  HCS  System 

Not  a  hi^  enough  priority.  Funding  not  available. 

Figure  4;  Technologies  Droj^jed  from  Furtiier  Consideration 
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2.4.2 


Toxicity  Response  Test  Plan  Developed  and  Implemented. 


After  the  selection  of  the  1 1  ESB  technologies  for  additional  testing  was 
completed,  a  twelfth  technology  was  identified  and  approved  for  inclusion  into  the  study;  the 
Hepatocyte  Low  Density  Lipoprotein  Uptake.  This  technology  was  included  partly  because  the 
toxicity  response  tests  were  provided  at  no  additional  cost. 

The  twelve  technologies  listed  in  Figure  5  are  ones  the  technical  experts 
determined  were  the  most  promising  and  for  which  toxicity  response  data  was  collected.  These 
technologies  were  later  assessed  with  the  quantitative  models. 

As  stated  earlier,  toxicity  response  results  were  needed  to  further  evaluate  each 
technology.  To  acquire  this  data  for  the  12  ESB  technologies  an  experimental  design  and 
analysis  Toxicity  Response  Test  Plan  was  developed. 

Part  of  this  experimental  plan  was  the  requirement  to  identify  and  select  a  set  of 
training/test  chemicals;  the  technical  experts  did  this  wiA  input  fi'om  the  user  representatives. 
These  chemicals  were  used  to  form  a  common  basis  for  comparison  of  test  results  for  the  ESB 
technologies.  Twelve  chemicals  were  selected  to  be  the  training/test  chemical  set  (see  Appendix 
H  for  a  more  detailed  discussion  on  the  selection  of  test  chemicals  and  the  list  of  chemicals). 
Battelle  Memorial  Institute  then  used  the  Toxicity  Response  Test  Plan  to  conduct  testing  and 
determine  the  toxicity  response  for  the  12  ESB  technologies.  These  results  have  been 
published.**  Note  that  for  the  Hepatocyte  Low  Density  Lipoprotein  Uptake  technology  the  test 
results  were  provided  by  the  vendor  and  were  not  independently  verified  by  Battelle. 


Olson,  C.;  James,  R.;  Botsford,  J.;  Curtis,  T.;  Doherty,  F.;  Lush,  D.;  McFadden,  P.; 
O’Shaughnessy,  T.;  States,  S.;  Shoji,  R.  A  Medical  Research  and  Evaluation  Facility  and  Studies 
Supporting  the  Medical  Chemical  Defense  Program;  Subtitle  -  Experimental  Design, 
Coordination,  and  Comparative  Analysis  of  Toxicity  Sensor  Data  Study',  Battelle  Memorial 
Institute;  Columbus,  OH,  2005;  UNCLASSIFIED  Report  (AD-A434473). 
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ESB  Technology  Name 

Abbreviated  ESB  Technology  Name 
(used  in  some  figures  and  tables) 

Eclox 

Eclox 

Microtox 

Microtox 

Mitoscan  Electron  Transfer  (ETR) 

Mitoscan  ETR 

Mitoscan  Reverse  Electron  Transfer  (RET) 

Mitoscan  RET 

ToxScreen  Metal 

ToxScreen  Metal 

ToxScreen  Organic 

ToxScreen  Org 

SinorhizobiiM  meliloti  Assay 

S.  meliloti 

SOS  Cytosensor 

SOS  Cyto 

Neuronal  Microelectrode  Array 

Neur  Microel  Array 

Toxi-Chromotest 

T  oxi-Chromotest 

Electric  Cell-Substrate  Impedance  Sensing 

ECIS 

Hepatocyte  Low  Density  Lipoprotein  (LDL) 
Uptake 

Hepatocyte  LDL 

Figure  5:  Technologies  Furtha-  Tested  and  Evahiated 


2.5  Quantitative  Assessment  Models  Described. 

Based  on  the  Toxicity  Response  Test  results,  the  technical  experts  determined  that 
no  ESB  technology  or  system  of  technologies  could  meet  minimal/threshold  requirements  for  the 
continuous  flow  scenarios  or  the  Individual  Soldier/Small  Team  scenario.  User  representatives, 
recommended  these  scenarios  not  be  considered  further  in  this  downselection  process,  but  their 
requirements  continue  to  be  incorporated  into  the  ESB  system  objective  requirements  for 
implementation  in  the  future  as  ESB  system  technologies  mature.  Therefore,  only  the  Fixed 
Base,  Field/Contingency,  and  Small  Unit  scenarios  (grab  sample  collection  method  only)  were 
considered  further.  Three  separate  multi-criteria  decision-making  models  were  developed. 

A  decision  support  software  tool.  Logical  Decisions  for  Windows  (LDW;  Lo^cal 
Decisions,  Inc.,  copyri^t  2000),  was  used  to  develop  and  document  the  downselection 
assessment  models. 

Assessment  criteria  are  the  core  of  the  models.  The  criteria  are  structured  as  a 
hierarchical  model  and  are  at  a  level  that  permits  discrimination  between  the  different 
technologies.  User  representatives  created  assessment  criteria  for  each  model  fi'om  the 
requirements  for  the  three  scenarios  previously  defined  (see  Appendix  B).  Higjier-level 
measures  categories,  referred  to  as  goals,  are  Performance,  Operational  Impact,  and  Logistics. 
Specific  sub-categories  of  criteria  (e.g..  Chemical  Detection),  referred  to  as  measures,  are 
developed  to  provide  the  degree  of  discrimination  needed  for  the  technology  evaluation. 
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Each  ESB  technology  was  assessed  against  these  measures.  Each  model  is 
comprised  of  ten  measures.  Measures  are  composed  of  definitions,  performance  scales,  and 
weights.  Several  factors  were  considered  when  developing  the  assessment  measures. 
Assessment  measures  need  to  provide  differentiation  between  the  ESB  technologies,  so  the 
measures  had  to  be  relevant  and  discriminating.  Measures  also  had  to  be  independent,  so  that 
aspects  measured  in  one  measure  were  not  repeated  in  another  measure.  It  was  likewise 
important  to  focus  on  the  measures  that  were  most  critical  to  the  analytical  process. 

Measures  can  define  quantitative  and/or  qualitative  requirements  or  goals.  For 
example,  the  Chemical  Detection  measure  was  quantitative,  measured  in  numerical  units  - 
number  of  chemicals  detected.  The  Storage  Conditions  Required  measure,  which  was 
qualitative,  was  better  assessed  in  more  subjective  terms,  such  as  “demanding  (refiigeration 
required).” 


Measure  definitions  and  performance  scales  are  important  elements  in  describing 
a  measure.  Measure  definitions  are  narrative  descriptions  of  the  measures  that  must  be 
adequately  and  appropriately  stated  and  clearly  understood.  Performance  scales  are  the  “rating 
scheme”  used  to  evaluate  technologies  against  the  measure.  Some  performance  scales  may  be 
continuous  (e.g.,  numeric  range  with  Chemical  Detection  measure),  while  others  may  be 
discontinuous,  or  discrete  levels  referred  to  as  labels  (e.g.,  levels  of  storage  conditions  required 
with  Storage  Conditions  Required  measure).  These  two  examples  are  shown  below. 

Chemical  Detection 

Utility  Performance  Scale 

100  Six  chemicals  detected 

0  No  chemicals  detected 

Storage  Conditions  Required 

Utility  Performance  Scale 

100  Easy,  ambient  conditions 

50  Average,  controlled  room  temperature 

0  Demanding,  refiigeration  required 

Performance  scales  are  expressed  as  utility  functions,  which  convert  different 
measures  to  common  units.  In  order  to  set  relevant  endpoints  and  to  establish  appropriate 
intermediate  utility  values,  ESB  technology  characteristics  had  to  be  well  defined.  Utility  values 
of  100  and  0  were  assigned  to  the  high  and  low  end  of  each  performance  scale  and  intermediate 
level  utilities  were  derived  through  various  elicitation  techniques  focused  on  the  relative 
importance  of  moving  to-and-firom  various  points  on  the  utility  function. 

Figure  6  illustrates  the  intermediate  utility  points,  in  the  form  of  a  utility  curve, 
for  the  Test  Reproducibility  measure.  This  utility  curve  is  referred  to  as  a  “risk  seeking”  curve; 
where  the  rate  of  utility  increases  rapidly  as  the  desired  end  of  the  scale  (6%)  is  approached. 
Utility  can  also  be  defined  by  risk  averse  and  linear  curves. 
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F^;ure6:  Measure  UtiHtyC^e  Example 

The  final  model  development  step  was  to  weight  the  measures,  based  on  their 
importance  relative  to  the  other  measures.  Measure  weighting  considers  both  relative  priority 
and  the  concept  of  swing  wei^ting.  Swing  weighting  compares  the  effects  of  movement  fi^om 
the  lowest  point  on  the  performance  scale  to  the  highest  for  any  measure  in  relation  to  a  similar 
move  for  any  other  measure.  An  example  would  be  determining  whether  it  was  more  important 
to  move  fi'om  “6%”  to  “54%”  median  coefficient  of  variation  for  the  Test  Reproducibility 
measure  as  compared  to  moving  finm  “easy”  to  “demanding”  for  the  Storage  Conditions 
Required  measure. 

Several  weighting  techniques  (Smarter  Method,  Analytical  Hierarchy  Process 
[AHP]  and  Direct  Entry  Assessment)  were  utilized  to  facilitate  the  development  of  the 
assessment  model  weights.  The  Smarter  Method  and  the  AHP  weighting  techniques  were  used 
as  a  starting  point  to  establish  measures  weights,  after  which  the  measures  weights  were  adjusted 
as  necessary  with  the  Direct  Assessment  technique,  as  this  technique  allows  for  a  simple  direct 
entry  of  weights. 

The  DAT  developed  a  draft  assessment  model  for  the  Fixed  Base  scenario  based 
on  the  already  defined  technical  requirements  (see  Appendix  BV  This  model  was  reviewed  and 
modified  by  the  user  representatives  with  input  fi’om  the  technical  experts.  The  user 
representatives  then  created  an  assessment  model  for  the  Field/Contingency  and  Small  Unit 
Scenarios.  The  Fixed  Base  Scenario  model  is  presented  in  Figure  7  (see  Appendix  I  for  Small 
Unit  and  Field/Contingency  Scenario  models  and  for  a  description  of  model  components  for  all 
three  models). 
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Figure  7:  Fixed  Base  Scenario  Model 

Figure  8  shows  the  definitions  for  all  ten  measures,  which  are  the  same  for  the 
three  models.  Figure  9  lists  the  performance  scales  plus  weights  for  the  three  models  and  shows 
five  of  the  ten  measures  differ  in  performance  scales  or  utility  functions.  For  example,  for  the 
Biological  Component  Maintenance  measure,  a  technology  requiring  “some”  biological 
component  maintenance  would  score  zero  utility  in  the  Small  Unit  Scenario  model  but  a  60% 
utility  in  the  Fixed  Base  scenario  model.  Nine  of  the  ten  measure  weights  are  also  different  for 
each  model.  For  example,  the  Test  Reproducibility  measure  is  weighted  23%  in  the  Fixed  Base 
Scenario  model,  13%  in  the  Field/Contingency  Scenario  model,  and  9%  in  the  Small  Unit 
,  Scenario  model  (see  Figure  9  below).  Only  the  Chemical  Detection  measure  has  the  same 
weight  in  all  models.  Note  this  measure  is  worth  almost  50%  of  each  model’s  overall  weight. 
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Measure  Definitions  for  each  Scenario/Mbdel 

Measure 

Definition 

Chemical 

Detection 

The  ability  of  the  technology  to  provide  an  adverse  response  to  a  set  of  representative 
chemicals  at  concentrations  between  the  short-term  MEG  and  the  HLC  for  the  greatest 
number  of  test  chemicals.  For  this  measure,  it  does  not  matter  how  close  the  technology 
provides  an  adverse  response  to  the  short-term  MEG. 

Water 

Characteristics 

The  ability  to  operate  under  a  number  of  water  quality  conditions  and  in  the  presence  of 
interfering  substances  with  minimal  effect  on  test  outcome.  The  ability  to  operate  in  water 
containing  residual  chlorine  is  a  surrogate  for  this  measure. 

Technology  Failure 
Rate 

The  number  of  times  a  technology  foils  to  produce  viable/usable  results  per  a  certain 
number  of  tests.  Excluded  from  this  measure  are  foilures  caused  by  human  mistake.  For 
each  technology  evaluated  the  technology  foilure  rate  includes  the  biological  basis  of  the 
system  and  all  ancillary  equipment. 

Test 

Reproducibility 

The  ability  of  the  ESB  technology  to  produce  the  same  output/answer  given  the  same  input 
and  test  conditions  over  multiple  tests.  This  is  measured  as  a  median  coefficient  of 
variation;  a  low  coefficient  of  variation  represents  high  reproducibility. 

Test  Turn  Around 
Time 

The  time  required  between  consecutive  tests  (assumes  reus^le  technology).  Includes 
operator  set-up  time,  sample  preparation,  technology  operation  time,  and  any  time  required 
for  the  system  to  reset  before  another  reading. 

Sample  Preparation 
&  S^lls  Required 

The  number  of  steps  (few  vs.  many)  and  complexity  of  steps  (e.g.,  measuring  volume, 
adding  reagent)  to  perform  tests  along  with  level  if  skills  and  focilities  requited  (at  sample 
site  vs.  in  a  lab)  to  complete  tests.  The  focus  is  not  on  how  long  it  takes  to  perform  each 
task  but  rather  how  complicated  each  task  is  and  what  level  of  skills  are  required  to 
conq)lete  the  tasks. 

Weight 

The  weight  of  the  technology  and  necessary  peripheral  equipment  (e.g.,  maintenance  and 
repair  supplies  and  parts,  power  source,  and  a  2-week  supply  of  consumables). 

Shelf  Life 

Length  of  storage  life  for  the  technology  and  consumables  (i.e.,  how  long  will  technology 
be  usable  [includes  shipping  time]  under  optimal  conditions  for  that  technology). 

The  conditions  required  to  maintain  tiie  technology  in  operating  condition  for  the 
maximum  possible  shelf  life. 

rime  required,  con^lexity  of  system,  and  materiel  and  logistics  required  to  maintain 
biological  component  of  foe  test  system. 

Figure  8;  Definitions  ofMeasuresfiir  all  Models 
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Measure  Performance  Scales  and  Weights  for  each  Scenario/Model 


Scenarios/Models 


Fixed  Base  I  Field  Contingencv  Small  Unit 


Performance  Scale  I  Wet  I  Performance  Scale 


Performance  Scale 


100  =  6  chemicals 
0  =  0  chemicals 
Linear  continuous  curve 


9 

100  =  6  chemicals 

0  =  0  chemicals 

100  =  6  chemicals 

0  =  0  chemicals 

i 

lin^  continuous  curve 

9  j 

Linear  continuous  curve 

1 

echnology 
Failure  Rate 


100  =  0%  Mures 
67  =  >0-5% 

33  =  >5-15% 

0  =  >15% 
Discrete  levels 


100  =  5  minutes 
0  =  180  minutes 
Nonlinear  continuous  curve 
same  for  all  scenarios) 


100  =  few,  minimal 
67  =  some,  moderate 
33  =  many,  significant 
0  =  many,  expert 
Linear  continuous  curve 


100=  11b 
0  =  401b 

S-shaped  continuous  curve 
different  for  each  scenario) 


100=  >12  mo 
)=<0.5mo 

onlinear  continuous  curve 


100=Easy 
50=Average 
30=Demanding 
Extremely  Demanding 
scrcte  levels 


100  =  None 
60  =  Some 

0  =  Media  exchange  or  C02 

required 

Discrete  levels 


100  =  >5mg/L 
75  =  >2-5mg/L 
50=  l-2mg/L 
0  =  <lmg/L 
Discrete  levels 


100  =  0%  Mures 
80  =  >0-5% 

30  =  >5-15% 

0  =  >15% 
Discrete  levels 


100  =  6%  median  coefficient  of 
variation 

0  =  54%  13 

Nonlinear  continuous  curve 
(same  for  all  scenarios) 


100  =  5  minutes 
0  =  180  minutes 
Nonlinear  continuous  curve 
(same  for  all  scenarios) 


100  =  few,  no  skills 
90  =  few,  minimal 
50  =  some,  modwate 
0  =  many,  significant 
linear  continuous  curve 


100=  Hb 
0  =  401b 

S-shaped  continuous  curve 
different  for  each  scenario) 


100=Easy 
j  50=Average 
0=Demandmg 
Discrete  levels 


I 


100  =  None 
=  Some 

=  Media  exchange  or  C02 
required 
Discrete  levels 


100  =  few,  no  skills 
70  =  few,  minimal 
0  =  some,  moderate 
linear  continuous  curve 


100=  lib 
0  =  401b 

S-shaped  continuous  curve 
(different  for  each  scenario) 


100=  >12  mo 
75  =  6-12  UK) 
30  =  3-6  mo 
0=<  3  mo 
Discrete  levels 


100=Easy 
0=Controlled 
Discrete  levels 


16 


Programmatic  factors/measures  such  as  cost,  risk,  and  potential  for  the  ESB 
technology's  capabilities  to  improve  are  not  included  in  the  models,  lliese  decision  factors,  though, 
were  considered  along  with  the  model’s  results  in  the  final  conclusions  and  recommendations. 
Technical  experts  provided  important  insights  and  data  on  these  factors  for  the  most  promising  ESB 
technologies  (see  Appendix!  for  programmatic  information  for  top-ranked  and  recommended 
technologies).  Figure  70  is  an  example  ESB  technology  programmatic  assessment  Programmatic 
input/comments  are  very  general  because  of  the  limited  programmatic  data  about  each  technology 
(most  technologies  are  in  an  early  stage  of  development)  and  because  of  the  limited  time  the 
technical  experts  had  to  complete  this  assessment. 


Programmatics  /^sessment 

ESB 

Technology 

Cost  (see  Note  1, 2,  &  3) 

Risk  (see  Note  1  &  3)) 

Potential  for  Technology's 
Capabilities  to  Improve  (see  Note 
1&3) 

Fairly  reasonable  for  capital 
investment  and  maintenance. 
Some  increased  costs  due  to 
need  to  maintain  cultures. 

Low  risk.  Very  mature 
technology. 

Litde  potential  for  further 
improvement 

Note  1 :  Because  technologies  are  in  an  early  stage  of  development,  because  of  die  limited  programmatic  data 
about  each  technology,  and  because  of  the  limited  time  the  technical  experts  had  to  complete  this  assessment 
the  programmatic  ii^ut/comments  are  veiy  gencral. 

Note  2:  Costs  don’t  include  initial  technology  development  costs  but  rather  only  costs  to  initially  buy  final 
technology  product  and  maintain  it  to  include  supplies  to  operate  it. 

Note  3:  Maturity  level  of  technology  is  considered  in  evaluation. 

Figure  10:  Exan:pleofTedhtx)lpgyPn)grammaticsAssessmait 
2.6  Analysis  of  Results. 

The  DAT  facilitated  a  two-day  meeting  in  May  2005  where  the  technical  experts 
completed  the  quantitative  downselect  assessment.  Technical  experts  used  a  combination  of  both 
quantitative  and  qualitative  data  about  the  ESB  technologies  along  with  their  knowledge  of  similar 
technologies  and  mechanisms.  Final  scores  were  a  consensus  of  expert  judgment,  not  an  average  of 
the  individual  technical  expert  assessments.  Assumptions  and  rationales  for  scores  were  documented 
where  necessaiy. 

Overall  scores  were  calculated  for  each  technology  by  multiplying  the  numerical 
rating  fi-om  the  performance  scale  by  the  weight  of  each  measure,  and  summing  the  scores  over  all 
measures  (linear  additive  fimction).  The  highest  possible  score  would  be  100.  While  the  overall 
scores  are  important,  they  are  only  used  as  a  guide  to  formulate  findings  and  conclusions. 

The  analyses  of  results  were  completed  by  the  DAT  fix>m  three  different  perspectives. 
First,  the  overall  results  were  examined  to  discern  general  outcomes  and  trends.  Second,  each 
technology  was  examined  to  identify  strengths  and  weaknesses.  Third,  the  results  specific  to  each 
measure  were  examined  to  identify  potential  technical  obstacles  or  shortcomings.  This  examination 
included  performing  sensitivity  analysis  to  detennine  what  effect  varied  measure  weights  would  have 
on  the  results.  LDW  was  used  in  the  analysis  because  it  has  many  useful  tools  for  perfonning 
comparative  analysis  and  displaying  and  dociunenting  results. 
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2.6.1 


Overall  Results. 


Figures  11-16  show  tlie  overall  score  and  rank  of  each  ESB  technology  for  the 
three  models.  The  figures  also  show  where  each  technology  is  strong  and  weak,  relative  to  the 
three  goals  and  ten  measures  of  the  model.  The  first  figure  for  each  scenario  (see  Figures  11, 13, 
15)  presents  the  overall  results  by  showing  the  results  of  how  each  technology  performed  against 
the  three  goals.  The  second  figure  for  each  scenario  (see  Figures  12,  14, 16)  shows  similar 
information  as  what  the  first  figure  presents,  but  presents  information  in  a  greater  level  of  detail 
by  showing  how  each  technology  scored  against  the  ten  measures.  The  length  of  each  of  the  sub¬ 
bars  indicates  how  much  of  the  technology’s  overall  score  is  attributable  to  the  three  goals  or  ten 
measures  (note  the  pattern  of  a  bar  in  the  stack  bar  chart  may  change  fi’om  model  to  model  [e.g., 
the  Shelf  Life  measure  bar  pattern  is  different  in  Figures  12  and  14f). 

2.6. 1.1  Fixed  Base  Scenario. 

In  the  Fixed  Base  Scenario  model  {Figures  11-12  show  results)  the  ESB 
technologies  can  be  categorized  into  five  broad  ranking  tiers:  significantly  above  average 
(Microtox),  above  average  (Hepatocyte  Low  Density  Lipoprotein  Uptake,  Electric  Cell-Substrate 
Impedance  Sensing,  Mitoscan  Electron  Transfer),  average  (Eclox,  Neuronal  Microelectrode 
Array,  SOS  Cytosensor,  Toxi-Chromotest),  below  average  (ToxScreen  Organic,  ToxScreen 
Metal,  Sinorhizobium  meliloti  Assay),  and  significantly  below  average  (Mitoscan  Reverse 
Electron  Transfer). 

Figure  11  illustrates  the  overall  technology  scores  and  ranking  relative  to  the 
three  model  goals.  This  figure  indicates  that  the  Performance  goal  is  the  strongest  discriminator, 
primarily  due  to  its  high  weight  (79%;  refer  to  Figure  9  for  the  list  of  weights  for  each  measure 
in  each  model),  but  also  due  to  the  differing  scores  among  most  of  the  ESB  technologies.  The 
Logistics  goal  has  a  lesser  effect,  mainly  because  of  its  lower  weight  in  the  model  (12%).  The 
Logistics  goal  provides  the  most  differentiation  among  the  top  three  technologies  as  they  score 
similarly  against  the  Performance  goal.  The  Operational  Impact  was  the  weakest  discriminator 
partly  because  this  was  the  least  weighted  goal  (9%). 

Figure  12  illustrates  the  overall  technology  scores  and  ranking  relative  to  the  ten 
model  measures.  This  figure  provides  a  greater  level  of  detail  for  each  technology  because  it 
provides  data  for  each  of  the  ten  measxires  used  in  the  model.  For  example,  the  figure  shows  that 
the  SOS  Cytosensor  received  about  80%  of  its  overall  score  fi-om  one  measure.  Chemical 
Detection,  while  Neuronal  Microelectrode  Array  scored  almost  95%  of  its  overall  score  fi-om  two 
measures  {Chemical  Detection  and  Test  Reproducibility).  This  figure  also  shows,  for  example. 
Chemical  Detection  and  Test  Reproducibility  measure,  the  two  highest  weighted  measures,  did 
not  provide  much  discrimination  between  Microtox,  Hepatocyte  Low  Density  Lipoprotein 
Uptake,  and  Electric  Cell-Substrate  Impedance  Sensing. 
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2.6. 1.2 


Field/Contineencv  Scenario. 


In  the  Field/Contingency  Scenario  model  {Figures  13-14  show  results)  the  ESB 
technologies  can  be  categorized  into  the  same  five  broad  ranking  tiers  as  the  Fixed  Base  Scenario 
model:  significantly  above  average  (Microtox),  above  average  (Hepatocyte  Low  Density 
Lipoprotein  Uptake),  average  (Electric  Cell-Substrate  Impedance  Sensing,  Mitoscan  Electron 
Transfer,  Eclox,  SOS  Cytosensor),  below  average  (Toxi-Chromotest,  Neuronal  Microelectrode 
Array,  ToxScreen  Organic,  ToxScreen  Metal,  Sinorhizobium  meliloti  Assay),  and  significantly 
below  average  (Mitoscan  Reverse  Electron  Transfer). 

In  this  scenario,  the  Operational  Impact  and  Logistics  goals  are  more  important 
(31%  of  model’s  weight;  refer  to  Figure  9  for  the  list  of  weights  for  each  measure  in  each  model) 
than  in  the  Fixed  Base  Scenario  (21%).  Most  ESB  technologies  dropped  in  score,  because  of  the 
relatively  higher  Operational  Impact  and  Logistics  goal  weights,  versus  how  they  scored  in  the 
Fixed  Base  Scenario  model.  The  exceptions  are  the  top  scoring  technology  (Microtox)  and  the 
lowest  scoring  technology  (Mitoscan  Reverse  Electron  Transfer),  which  scored  the  same,  and 
Eclox  and  SOS  Cytosensor,  which  have  slightly  higher  scores. 

The  technologies  scored  overall  in  the  same  ranking  tiers  as  in  the  Fixed  Base 
Scenario  model  except  for  Mitoscan  Electron  Transfer  which  dropped  from  the  above  average 
tier  to  the  average  tier  and  Toxi-Chromotest  and  Neuronal  Microelectrode  Array  which  dropped 
fi'om  the  average  tier  to  the  below  average  tier.  Microtox  is  still  the  best  overall  performer. 

T  echnology 
M  icrotox 
H  epatocyte  LD  L 
ECIS 

M  itoscan  ETR 
Eclox 
SOS  Cyto 
Toxi-Chromotest 
Neur  M  icroel  Array 
T oxScreen  O  rg 
T oxS creen  M  etal 
S  m  elitoti 
M  itoscanR E T 


■i  PERFORMANCE  Ei  OPERATIONAL  IMPACT  m  LOGISTICS 

Figure  13:  Overall  Assessment  Goal  Scores  Sir  ESB  Technologies;  Field/Contingency  Scenario 


20 


IT 


T  echnology 

S  core 

M  icrotox 

82 

Hepatocyte  LDL 

65 

ECIS 

60 

M  itoscan  ETR 

58 

Eclox 

56 

S  0  S  C  yto 

55 

T oxi-Chrom  otest 

46 

Neur  M  icroel  A  rray 

44 

ToxScreen  Org 

44 

Tox S creen  M  etal 

44 

S  m  elitot! 

43 

M  itoscanRET 

33 

■i  Chem  Detection 
ill  Shelf  Life 
W  Bio  Comp  Maint  S  H20  Char  ^Weight 

[TTl  Store  Cond 


iiiii  Reproduciblty  gni  Turn  Around 
EHi  Prep  &  Skills  m  Techn  Fail 


Figure  14;  Overall  Assessment  Measures  Scores  for  ESBTechnolo^es:Field/Conliiigeii(^  Scenario 
2.6. 1.3  Small  Unit  Scenario. 

In  the  Small  Unit  Scenario  model  (Figures  15-16  show  results)  the  ESB 
technologies  can  be  categorized  into  the  same  five  broad  ranking  tiers  as  the  other  models: 
significantly  above  average  (Microtox),  above  average  (Hepatocyte  Low  Density  Lipoprotein 
Uptake),  average  (Electric  Cell-Substrate  Impedance  Sensing,  Eclox,  Mitoscan  Electron 
Transfer,  SOS  Cytosensor),  below  average  (Toxi-Chromotest,  Neuronal  Microelectrode  Array, 
ToxScreen  Organic,  ToxScreen  Metal,  Sinorhizobium  meliloti  Assay),  and  significantly  below 
average  (Mitoscan  Reverse  Electron  Transfer). 

In  this  scenario,  the  Operational  Impact  and  Logistics  goals  are  even  more 
important  (3 1  %  of  model’s  weight;  refer  to  Figure  9  for  the  list  of  weigjhts  for  each  measure  in 
each  model)  than  in  the  Field/Contingency  (31%)  and  the  Fixed  Base  (21%)  Scenario  models. 
All  ESB  technologies  stayed  in  the  same  ranking  tier  as  the  Field/Contingency  Scenario  model., 
althou^  all  technology’s  scores  dropped.  The  drop  in  score  is  because  of  the  relatively  hi^er 
»  Operational  Impact  and  Logistics  goals,  versus  how  they  scored  in  the  Field/Contingency  or 

Fixed  Base  Scenario  models.  Microtox  is  still  the  hipest  scoring  technology  by  a  wide  margin, 
but  fell  the  most  in  overall  score.  Microtox’s  scored  dropped  from  82  to  76.  Figure  15  shows 
that  Mitoscan  earns  about  the  same  overall  score  with  the  Performance  goal  as  Eclox  earns  with 
two  goals  (Performance  and  Logistics). 
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Microtox  (82)  scored  above  average  on  six  measures  {Chemical  Detection,  Technology 
Failure  Rate,  Test  Turn  Around  Time,  Preparation  and  Skills  Required,  Shelf  Life,  and 
Biological  Component  Maintenance)  and  below  average  on  Water  Characteristics, 
Weight,  and  Storage  Conditions  Required  measures.  Microtox  detected  six  chemicals, 
including  one  chemical  the  other  three  top-ranked  technologies  did  not  detect  - 
cyanide.  _ 


Ffeurel?;  Exan:q)leofa  Technology  StTQttgflis  and  Weaknesses  Analysis  Nanative  Summary 

LDW  was  used  to  generate  each  technology’s  strengths  and  weaknesses  analysis 
chart.  The  charts  graphically  show  a  particular  technology’s  strengttis  and  weaknesses,  relative 
to  each  measures  weights.  Figure  18  is  an  example  of  a  technology’s  strengths  and  weaknesses 
analysis  chart.  The  height  of  the  bars  indicates  a  technology’s  relative  score  for  each  measure, 
while  the  width  indicates  the  relative  weight  of  the  measure.  This  chart  shows  that  Microtox 
scores  very  high  against  four  measures.  Chemical  Detection  (highest  wei^ted  measure  also). 
Shelf  Life,  Biological  Component  Maintenance,  and  Technology  Failure  Rate.  The  measures 
where  Microtox  scored  low  were  generally  weighted  low,  for  example  the  Water  Characteristics 
measure  (H20  Char,  6th  bar  from  the  left). 
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F^;ure  18:  Example  of  a  Technology  Sttengfhs  and  Weaknesses  Analysis  Chart 


Measure  Analysis. 


The  measures  analysis  summarizes  the  assessment  results  in  terms  of  individual 
measures  {Figure  19  is  an  example).  Because  the  measures  repr^ent  user  needs,  this  analysis 
helps  identify  areas  of  shortfall,  or  potential  technical  challenges  (e.g.,  if  the  ESB  technologies 
generally  score  low  against  a  measure).  Conversely,  the  analysis  also  identifies  areas  of  minimal 
concern  (e.g.,  the  technologies  genCTally  score  high  and/or  the  measure  is  low-weighted). 


The  range  of  scores  for  each  measure  was  examined  to  determine  overall  ESB 
technology  performance  relative  to  each  measure,  which  provided  the  basis  for  assignment  of  a 
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subjective  assessment  rating  by  the  DAT.  Comments  and  rationale  for  each  measure  rating  were 
also  provided.  Each  measure  was  assessed  on  a  green,  yellow,  red  scale  (see  Figure  19  for 
definition  of  rating).  All  measures  were  rated  either  green  or  yellow  -  no  measure  was  rated  red. 
Reference  Appendix  L  for  the  assessment/rating  and  Appendix  M  for  the  sensitivity  charts  for  all 
measures. 


Measures 

(Wgt) 

Rating  (see 
Rating  Key) 

Comments 

Chemical 

Detection 

(0.49) 

Green  (see 
Note  1) 

-  This  measure  is  weighted  the  highest,  almost  half  of  the  model’s 
wei^t. 

-  The  utility  function  is  a  linear  continuous  curve.  For  each 
chemical  detected  the  technology  earns  16.7%  of  the  possible 
score.  For  example,  detecting  four  chemicals  would  earn  a  score 
of  66.8  (4  x  16.7)  out  of  100. 

-  Seven  technologies  detected  three  chemicals  or  fewer  and  thus 
earned  half  or  less  of  the  possible  maximum  score. 

Note  1 :  Because  the  maximum  number  of  chemicals  detected  bv 
any  technology  is  six,  this  is  the  100  score,  but  it  is  a  goal  to 
detect  all  twelve  chemicals.  In  order  to  detect  more  than  six 
chemicals,  an  ESB  technology  “system”  is  needed  (i.e.,  two  or 
more  toxicity  sensors  woridng  together). 

Rating  Key: 

Green:  Area  of  low  concern  indicating  user  needs  should  be  met,  or  the  measure  is 
low  weight  and  is  unlikely  to  cause  significant  impact. 

Yellow:  Area  of  moderate  concern  indicating  user  needs  may  not  be  met. 

Red:  Area  of  high  concern  indicating  user  needs  are  not  readily  met. 

Figure  19:  Example  of  a  Measure  Analysis 

Sensitivity  analysis  was  also  performed  to  determine  whether  changes  in 
measures  wei^ts  would  affect  the  results.  This  analysis  was  conducted  on  all  measures.  LDW 
was  used  to  generate  sensitivity  graphs  for  each  measure.  Figure  20  shows  a  typical  example  of 
a  technology’s  sensitivity  analysis  chart.  The  sensitivity  graphs  contain  the  overall  score  on  the 
y-axis  (jfrom  worst  to  best)  and  the  percent  of  the  weight  given  to  the  measure  being  evaluated  on 
the  x-axis  (fi-om  0  to  100).  There  is  a  line  segment  for  each  ESB  technology  on  the  graph,  which 
extends  from  the  x-axis  at  various  angles  to  the  position  that  corresponds  with  100%  of  the 
weight  being  assigned  to  that  measure.  A  vertical  line  on  the  sensitivity  graph  identifies  the 
current  wei^t  assigned  to  the  measure.  The  order  in  which  the  vertical  weight  line  intersects  the 
alternatives’  segments  is  the  overall  rank  order  produced  by  the  model’s  wei^ting  scheme. 

By  visually  noting  the  changes  in  rank  order  which  would  occur  if  the  vertical 
weight  line  was  moved  to  the  right  or  left  (measure  weight  increased  or  decreased),  an 
assessment  can  be  made  as  to  the  sensitivity  of  the  weight  of  that  measure.  If  a  slight  movement 
of  the  weight  line  causes  multiple  alternative  lines  to  cross  (and  therefore  the  rank  order  to  have 
multiple  changes),  then  that  measure  would  be  considered  very  sensitive  to  wei^t  changes.  If, 
on  the  other  hand,  no  ranking  changes  occiu*  when  the  line  is  moved  a  greater  distance,  then  it 
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can  be  detennined  that  the  weight  of  that  measure  does  not  have  a  notable  effect  on  the  outcomes 
of  the  analysis. 


For  the  most  part,  the  technology  rankings  are  insensitive  to  reasonable  (±  15% 
changes)  measure  weight  changes.  This  is  especially  true  for  the  top  three  scoring  technologies 
(Microtox,  Hepatocyte  Low  Density  Lipoprotein  Uptake,  and  Electric  Cell-Substrate  Impedance 
Sensing).  There  are  two  instances  were  reasonable  weight  changes  have  an  impact: 

•  The  first  instance  is  when  the  Chemical  Detection  measure  wei^t  is  increased 
or  reduced  15%  (considered  maximum  reasonable  change;  see  Figure  20).  By 
decreasing  the  weight  of  the  Chemical  Detection  measure  15%  Mitoscan 
Electron  Transfer  and  Eclox  change  positions  in  the  ranking  (3”*  and  4* 
positions).  Toxi-Chromotest  increases  fi’om  8*  hipest  ranked  to  6*  hipest 
ranked.  By  increasing  the  wei^t  of  the  Chemical  Detection  measure  15%, 
the  impact  on  the  ranking  is  a  little  less  as  only  SOS  Cytosensor  increases 
from  7*  highest  ranked  to  5*  highest  ranked  and  exchanges  ranking  with 
Eclox. 

•  The  second  instance  is  when  the  wei^t  of  the  next  hipest  weighted  measure. 
Test  Reproducibility,  decreases  15%  (See  Appendix  M.  Figure  M-4).  When 
this  happens,  SOS  Cytosensor  increases  fipm  seventh  hipest  ranked  to  fifth 
highest  ranked.  Eclox  and  Neuronal  Microelectrode  Array  each  drop  one 
position  in  ranking.  When  the  measure  is  increased  15%  there  is  only  a  slight 
impact  -  Electric  Cell-Substrate  Impedance  Sensing  and  Mitoscan  Electron 
Transfer  woxild  rank  the  same  and  SOS  Cytosensor  and  Toxi-Chromotest 
would  rank  the  same. 


Percent  of  Weight  on  Chemical  Detection  Measure 
for  the  Fixed  Base  Scenario 


Microtox 
Hepatocyte  LDL 
ECIS 

Mitoscan  ETR 
Eclox 

Neur  Microel  Array 
SOS  Cyto 
Toxi-Chrom  Test 
ToxScreen  Org 
Tox Screen  Metal 
S  meliloti 
Mitoscan  RET 


Figure  20:  Example  ofa  Measure  Sensitivity  Analysis  Chart 
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3. 


CONCLUSIONS 


No  technology  detected  more  than  half  of  the  chemicals  required  within  the 
MEG-HLC  range.  All  chemicals  except  for  nicotine  were  detected  by  at  least  one  technology. 
Three  chemicals  were  only  detected  by  one  technology.  Only  Neuronal  Microelectrode  Array 
detects  aldicaih  and  methamidophos  and  only  Electric  Cell-Substrate  Impedance  Sensing  detects 
ammonia. 


By  creating  a  system  of  three  ESB  technologies  it  would  be  possible  to  detect  nine 
out  of  the  twelve  chemicals.  It  would  take  five  technologies  to  detect  all  twelve  chemicals. 

Even  the  best  three  ESB  technologies  only  scored  in  the  67-82  range  with  the 
Fixed  Base  Scenario  model  and  thus  have  areas  within  performance,  operational  impact,  or 
logistics  that  could  be  improved. 

ESB  technologies  that  scored  poorly  in  the  models  but  can  detect  an  important 
class  of  chemicals  may  be  recommended  for  fiuther  considered  and  research.  For  example,  the 
Daphnia  IQ  Test,  a  technology  eliminated  fix)m  consideration  with  the  qualitative  screening 
model,  is  the  only  technology  known  to  detect  nicotine  at  the  appropriate  sensitivity  level. 


4.  RECOMMENDATIONS 

Expand  the  ciurent  chemical  training/test  set  beyond  twelve  to  include  other 
classes  of  chemicals  and  additional  chemical  examples  for  the  current  classes  of  chemicals 
studied.  These  additional  chemicals  are  needed  to  enhance  the  project  team’s  understanding  and 
confidence  levels  of  toxicity  responses  for  the  ESB  technologies  chosen  for  fiirther  study  and 
development. 


Adopt  an  evolutionary  development  strategy  for  an  ESB  system.  Initially  focus 
on  the  Fixed  Base  Grab  Sample  Scenario  because  this  provides  the  greatest  chance  of  program 
success  (i.e.,  fielding  a  prototype  before  FY09).  Add  other  use  scenarios  in  future  program 
increments,  as  ESB  technologies  evolve  and  improve. 

Recommend  Microtox  (or  Deltatox),  Hepatocyte  Low  Density  Lipoprotein 
Uptake  Uptake,  Electric  Cell-Substrate  Impedance  Sensing,  Neuronal  Microelectrode  Array,  and 
Daphnia  IQ  be  further  researched  and  developed.  This  recommendation  is  based  on  the  fact  that 
Microtox  (or  Deltatox,  which  is  a  field  portable  version  of  Microtox),  Hepatocyte  Low  Density 
Lipoprotein  Uptake  Uptake,  and  Electric  Cell-Substrate  Impedance  Sensing  together  detect  nine 
different  chemicals  and  scored  the  highest  in  the  Fixed  Base  Scenario  model.  The 
recommendation  for  Neuronal  Microelectrode  Array  and  Daphnia  IQ  Test  is  because  they  detect 
three  chemicals  that  Microtox,  Hepatocyte  Low  Density  Lipoprotein  Uptake,  and  Electric  Cell- 
Substrate  Impedance  Sensing  do  not.  These  five  technologies  detect  all  twelve  chemicals.  Note 
that  Neuronal  Micoelectrode  Array  and  Daphnia  IQ  Test  scored  below  average  in  the  models  as 
they  both  have  logistical  and  operational  issues  that  must  be  addressed. 
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Use  the  results  firom  each  ESB  technology’s  measure  assessment  (see  Appendix 
L)  to  help  identify  areas  to  focus  research  and  development  (e.g.,  increase  Microtox’s  ability  to 
handle  varying  water  characteristics). 

Eclox  is  not  recommended  for  further  assessment  or  consideration.  This  is 
because  it  only  detects  one  chemical  (copper).  The  top  four  highest  scoring  ESB  technologies 
also  detect  copper  and  Eclox  detects  copper  much  closer  to  the  HLC  than  the  short-term  MEG. 
Eclox  rated  very  high,  though,  on  almost  all  measures  in  the  model  except  for  Chemical 
Detection,  Water  Characteristics,  and  the  Weight  measures.  The  technical  experts  especially 
liked  the  field  portability  and  maturity  level  of  Eclox. 

Althou^  the  initial  focus  of  the  ESB  ATO  may  be  to  deliver  an  ESB  prototype 
system  suitable  for  the  Fixed  Base  Scenario,  it  is  important  to  strive  to  achieve  the  requirements 
for  all  six  scenarios.  Seed  money  should  be  considered  for  some  ESB  technologies  that  may  not 
be  ready  for  near-term  fielding  but  that  may  be  the  best  option  long  term  for  scenarios  other  than 
the  Fixed  Base  Scenario. 

Examples  of  technologies  that  the  technical  experts  have  determined  don’t  have  a 
reasonable  potential  to  meet  the  current  ATO  requirranents  but  that  could  someday  possibly  meet 
the  threshold  reqixirements  for  the  other  five  scenarios  in  the  out  years  are  SOS  Cytosensor, 
Stressor-Specific  Escherichia  coli  Sensor,  Cell  Culture  Analog,  Multianaltye  Micro-  and  Nano- 
Physio-meters,  Caenorhabditis  elegans  Monitor,  and  Instant  Multicellular  Organisms.  These 
technologies  should  be  put  on  a  technology  watch  list.  The  technical  experts  also  recommended 
it  be  determined  if  a  well-developed  version  of  SOS  Cytosensor  already  exists. 

Resource  allocation  decision  support  methodologies  coxild  be  used  to  model  the 
various  funding  options  for  each  of  the  technologies  chosen.  This  would  allow  the  ESB  system 
program  management  to  perform  “what-if  ’  analysis  and  model  how  to  maximize  the  benefits 
while  addressing  potential  funding  cuts  or  additions. 

Decision  analysis  methodologies  and  tools  can  provide  a  fiamework  to  further 
analyze  current  and  new  ESB  technologies  as  additional  data  ^m  future  research  becomes 
available,  perhaps  as  part  of  a  structured  reassessment  process. 
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APPENDIX  A 


ENVIRONMENTAL  SENTINEL  BIOMONITOR  SYSTEM  DOWNSELECTION  PROJECT 

TEAM 

First  name 

Last  Name 

Affiliation 

Ric 

DeLeon 

Metropolitan  Water  District  of  Southern  California 

Jay 

Dusenbury 

US  Army  Research,  Development  and  Engineering 
Command 

Tom 

Gargan 

U.S.  Army  Center  for  Environmental  Health  Research 

Mike 

Goode 

Edgewood  Chemical  Biological  Center 

John 

Barbell 

Institute  for  In  Vitro  Sciences 

Wally 

Hayes 

Harvard  School  of  Public  Health 

Janet 

Jensen 

Edgewood  Chemical  Biological  Center 

Randy 

Kellar 

Boeing,  Inc.  (Representative  for  Lead  System 

Integrator,  Future  Combat  Systems  -  Manned  Groimd 
Vehicle  program) 

Scott 

Kooistra 

Edgewood  Chemical  Biological  Center 

Joe 

Pancrazio 

National  Institutes  of  Health  (Neurological  Disorders 
and  Stroke) 

Steve 

1  Richards 

U.S.  Army  Center  for  Healdi  Promotion  and 

Preventive  Medicine 

Robert 

Ryczak 

U.S.  Army  Center  for  Health  Promotion  and 

Preventive  Medicine 

Stanley 

States 

Pittsburgh  Water  &  Sewer  Authority 

Roy 

Thompson 

Edgewood  Chemical  Biological  Center 

Bill 

Van  der  Schalie 

U.S.  Army  Center  for  Environmental  Health  Research 

Jeremy 

Walker 

US  Army  Research,  Development  and  Engineering 
Command 

John 

Walther 

Edgewood  Chemical  Biological  Center 

Lindsey 

Wurster 

Edgewood  Chemical  Biolo^cal  Center 
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ENVIRONMENTAL  SENTINEL  BIOMONITOR  SYSTEM  TECHNICAL  REQUIREMENTS  FOR  USER  SCENARIOS 


neans  objective  or  ideal/prefeired. 

Scenarios  | 

Fixed  Base 

Test  Raw/Treated  Water 

c 

S 

I  SI 

C  c  f 

o  ^  ^ 
U  o 

E 

^  P 
w  C 

S  w 

4«« 

S'l 

<  A 

1 .  Operational: 
5-95%  RH  (T)2. 
Operational:  OC 
to  45C  (T);  -32C 
to  49C  (0)3. 
Storage:  OC  to 
52C  storage  (T); 
-10Cto71C(O) 

Must  function 
with  presence  of 
many  dissolved 
solids  up  to 
brackish  and 
turbidity  in  raw 
water  or 
chlorine  residual 
(2  ppm  (T),  5 
ppm  (O)  with 
treated  water 
(T). 

Grab  Sample 
Collection 
Method 

EPA(0)-MEG 
(O)- Acute  (T) 

1.  Operational:  5- 
95%  RH  (T)2. 
Operational:  OC  to 
45C  (T);  -32C  to 
49C  (0)3.  Storage: 
OC  to  52C  storage 
(T);-10Cto7lC 
(O) 

Must  function  with 
presence  of  many 
dissolved  solids  up 
to  brackish  and 
turbidity  in  raw 
water  or  chlorine 
residual  (2  ppm 
(T),5ppm(0) 
with  treated  water 
(T). 

Fleld/Contingency 

Test  Treated 
Water 

c 

<r  S 

1  C  5 

c  ^  ^ 
O  t  ^ 

u  I 

E 

MEG  >  short  + 
long  term  (0)- 
Acute,  i,e., 
MEG  -short 
tcrm(T) 

1 .  Operational: 
5-95%  RH  (T)2. 
Operational:  OC 
to  45C  (T);  - 
32C  to  49C 
(0)3.  Storage: 

OC  to  52C 
storage  (T):  - 
IOC  to71C(0) 

Must  function 
with  presence  of 
many  dissolved 
solids  up  to 
brackish  and 
turbidity  in  raw 
water  or 

chlorine  residual 
(2  ppm  (T),  5 
ppm  (0)  with 
treated  water 
(T). 

Test  Raw/  Treated 
Water 

Grab  Sample 
Collection  Method 

MEG  -  short  +  long 
term  (0)-Acutc, 
i.e.,  MEG  -short 
tcnn(T) 

1.  Operational:  5- 
95%  RH  (T)2. 
Operational:  OC  to 
45C  (T);  -32C  to 
49C  (0)3.  Storage: 
OC  to  52C  storage 
(T);-10Cto71C 
(O) 

Must  function  with 
presence  of  many 
dissolved  solids  up 
to  brackish  and 
turbidity  in  raw 
water  or  chlorine 
residual  (2  ppm 
(T),5ppm(0) 
with  treated  water 
(T). 

Small  Unit 

Test  Raw/Treated 
Water 

Grab  Sample 
Collection 
Method 

Acute,  i.e.  MEG- 
short  term  (T) 

1.  Operational;  5- 
95%  RH  (T)2. 
Operational:  OC  to 
45C  (T);  -32C  to 
49C  (0)3.  Storage: 
OC  to  52C  storage 
(T);-10Cto71C 
(O) 

Must  fiinction  with 
presence  of  many 
dissolved  solids  up 
to  brackish  and 
turbidity  in  raw 
water  or  chlorine 
residual  (2  ppm 
(T),5ppm(0) 
with  treated  water 
(T). 

Indhidual/Smail 

Team 

Test  Raw  Water 

Grab  Sample 
Collection  Method 

Acute,  i.e.  MEG- 
short  term  ( T) 

1.  Operational:  5- 
95%  RH  (T)2. 
Operational:  OC  to 
45C  (T):  -32C  to 
49C  (0)3.  Storage: 
OC  to  52C  storage 
(T);-lOCto7lC 
(O) 

Must  function  with 
presence  of  many 
dissolved  solids  up 
to  brackish  and 
turbidity  in  raw 
water  or  chlorine 
residual  (2  ppm  (T), 

5  ppm  (0)  with 
treated  water  (T). 

c 

i 

1 

1 

1 

£ 

‘c 

E 

O 

Tech  Requirements 

What's  the  level  of  detection 
for  each  chemical? 

Range  for  air  temperature, 
atmospheric 
pressure/altitudc,  and 
humidity  level? 

Other  water  characteristics 
device  must  handle  (e.g., 
quantity  of  total  dissolved 
solids,  turbidity,  chlorine 
residual)? 
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Assessment 

Category 

Detection 

Environmental 
Conditions 
During  Test 

Environmental 
Conditions 
During  Test 
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ENVIRONMENTAL  SENTINEL  BIOMONITOR  SYSTEM  TECHNICAL  REQUIREMENTS  FOR  USER  SCENARIOS 

T”  means  threshold  or  minimum  reauirement!  "O”  means  ohier.Hve  nr  ideal /nreferr^H  1 
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Scenarios  | 

Fixed  Base 

Test  Raw/Treated  Water 

Continuous 
Flow  Collection 
Method 

Ycs(T) 

24  hrs  (T),  168 
hrs  (O) 

30  days  (T),  60 
days  (O) 

30  days  (T),  1  yr 
(0) 

Grab  Sample 
Collection 
Method 

P 

(A 

V 

> 

One  test  (T),  1 50 
tests  (O) 

one  per  850  tests 
(T&O) 

30  days  (T),  1  yr 
(0) 

Fiekl/Contingency 

Test  Treated 
Water 

Continuous 
Flow  Collection 
Method 

Yes  {T) 

20  hrs  (T),  168 
hrs  (O) 

280  hrs  (20 
hrs/day  for  14 
days)(T) 

560  hrs  (20 
hrs/day  for  28 
days)  (O) 

P  O 

Test  Raw/  Treated 
Water 

Grab  Sample 
Collection  Method 

Yes  (T) 

One  test  (T),  1 50 
tests  (O) 

one  per  850  tests  (T 
&0) 

1  yr  (T),  2  yr  (0) 

Small  Unit 

Test  Raw/l'reatcd 
Water 

Grab  Sample 
Collection 
Method 

Yes  (T) 

One  test  (T),  1 50 
tests  (O) 

one  per  850  tests 
(T&O) 

1 

Individual/Smail 

Team 

Test  Raw  Water 

Grab  Sample 
Collection  Method 

Yes(T) 

One  test  (T),  1 50 
tests  (O) 

one  per  850  tests  (T 
&0) 

1 

Tech  Requirements 

Device  has  a  indicator  that 
ESB  system  is  still  working 
(or  has  a  malfunction 
signal)? 

Number  of  tests  (or  hours  if 
using  continuous  collection 
methexi)  before  reloading, 
recalibration? 

Reliability?  Device  failure 
rate  per  X  amount  of  tests  (or 
time  if  using  continuous 
collection  method)? 

Shelf  life  (device  and 
consTimables)? 

Assessment 

Category 

Robustness 

Robustness 

Robustness 

Robustness 

% 

VO 

M 

APPENDIX  B 


B-5 


_ .  _ environmental  sentinel  biomonitor  system  technical  requirements  for  user  scenarios 

Notes;  ”T"  means  threshold  or  minimum  requirement;  "O"  means  objective  or  ideal/preferred. 
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soldier/  single  or  depot  dius 
technician/  moderately 
complex)? 


APPENDIX  C 


ADDITIONAL  INFORMATION  ON  MILITARY  EXPOSURE  GUIDELINES 

Military  Exposure  Guidelines  (MEGs)  are  provided  for  water  (and  other  media)  to  estimate  a 
level  .  above  which  certain  types  of  health  effects  may  begin  to  occur  in  individuals  amongst 

the  exposed  population”.  MEGs  are  . .  designed  to  indicate  ‘thresholds’  for  minimal  to  no 
adverse  health  effects”  and  are  considered  .  protective  against  any  significant  non-cancer 
effects”.  MEG  exposure  scenarios  for  water  are  appropriate  for  a  dq)loyed  military  operation, 
i.e.,  exposure  lasting  either  5  or  14  days,  with  water  consumption  of  either  5  or  15  L/day.  The 
expos^  population  is  defined  to  include  “. . .  relatively  healfiiy  and  fit  male  and  non-pregnant 
female  adults”,  1 8  to  55  years  old  with  and  average  weight  of  70  kg.  It  is  important  to  note  that 
while  MEGs  are  not  enforceable  military  standards,  the  MEGs  are  considered  guideline 
concentrations  for  identifying  and  ranking  Occupational  and  Environmental  Health  (OEH)  risks. 
MEGs  have  been  established  for  about  190  chemicals.  MEGs  provide  a  reference  point  above 
which  adverse  effects  may  be  expected  after  a  field-relevant  period  of  exposure  and  may  serve  as 
lower  thresholds  for  toxicity  sensor  responses.  That  is,  responses  at  concentrations  below  the 
MEGs  indicate  toxic  effects  that  may  not  be  relevant  to  acute  human  health  impairments. 

Reference:  U.S.  Army  Center  for  Health  Promotion  and  Preventive  Medicine.  2001 .  Chemical 
Exposure  Guidelines  for  Deployed  Military  Personnel.  TG-230.  U.S.  Army  Center  for  Health 
Promotion  and  Preventive  Medicine,  Aberdeen  Proving  Ground,  MD. 
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CANDIDATE  ENVIRONMENTAL  SENTINEL  BIOMONITOR 
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EXAMPLE  OF  AN  ENVIRONMENTAL  SENTINTEL  BIOMONITOR  TECHNOLOGY  FACT 

SHEET 

Microtox,  Deltafox 


(1)  Technology  description 

a.  Vendor 

Strategic  Diagnostics  Inc. 

1 1 1  Pencader  Drive,  Newark  DE  19702-3322  USA 

Sales:  (800)  544-8881  Tel:  (302)  456-6789  Fax:  (302)  456-6782 

(http://www\sdix.com/ProductSpecs.asp?nProductID=7) 

b.  Toxicity  sensor  type 

i.  Living  system  used:  Vibrio  fischeri  (luminescent  bacteria) 

ii.  Endpoint  monitored:  Inhibition  of  light  output  or  toxicity  threshold;  light 
is  produced  as  a  byproduct  of  cellular  respiration. 

iii.  Monitoring  method 

1 .  Freeze-dried  bacteria  are  reconstituted  in  salt  solution  containing 
water  sample  to  be  tested 

2.  Luminescence  readings  are  taken  prior  to  and  at  5  and  15  min. 
after  water  sample  addition 

iv.  Continuous  monitoring  or  grab  sample?  Grab  sample 

V.  Note:  Microtox  and  Deltatox  generally  produce  similar  results;  Deltatox 
is  designed  to  be  field  portable  and  lacks  the  temperature  control 
capabilities  of  Microtox. 

(2)  Logistical  considerations 

a.  Environmental  requirements:  Testing  done  in  salt  water  solution;  temperature  is 
maintained  at  15  deg.  C;  ambient  temperatures  must  be  15-30  deg.  C. 

b.  Storage  and  transport:  Shelf  life  of  freeze-dried  bacteria  1  year  when  stored  at  -20 
to  -25°  C;  less  at  normal  refiigerator  temperatures. 

c.  Sample  throughput:  Microtox:  1 5  samples  per  hour;  Deltatox:  20  per  hour 

d.  Cube  and  weight:  Microtox:  1.0  ft^;  21  lbs;  Deltatox:  0.2  ft^,  6  lbs 

(3)  Response  to  toxic  chemicals  (EPA  ETV  report  available) 

a.  Chemicals  tested:  extensive  database;  see 
http.•//v^^^sdix.com/pdf7Toxicitv%20Data%20Index.pdf:  also  tested  as  part  of 
MEIC  in  vitro  cytotoxicity  evaluation  program;  see  graphs  on  next  page 

b.  Toxicity  data: 

i.  ETV  results:  no  inhibition  up  to  human  lethal  levels  for  botulinum  toxin, 
ricin,  soman,  and  VX;  aldicarb,  colchicine,  cyanide,  dicrotophos,  and 
thallium  were  detected  at  or  below  lethal  levels 

ii.  Response  relative  to  human  lethal  values  and  MEGs:  See  graph  on  next 
page 

c.  Test  precision:  Range  of  test  standard  deviations  from  ETV  evaluation  were  most 
less  than  10%. 

d.  Interfering  substances  and  water  conditions: 


E-1 


i.  Samples  must  be  dechlorinated  before  testing;  chloraminated  water  may 
interfere  after  dechlorination 

ii.  Tuibid  and  colored  samples  may  require  pre-treatment;  pH  outside  range 
6.0  -  8.0  requires  adjustment 

iii.  EW  testing  showed  copper  and  zinc  to  cause  interferences 
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Microtox  Toxicity  Data 
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Lethal  doses  for  50  chemicals  (Wallum,  1998)  are  plotted  in  order  of  increasing  toxicity  as 
concentrations  in  water,  assuming  a  70  kg  person  consuming  5  L  of  water.  Military  Exposure 
Guidelines  (MEGs)  for  water  (5  day,  5  L/day)  are  available  for  15  of  the  50  chemicals 
(USACHPPM,  2001).  Microtox  data  from  K^ser  and  Palabrica,  1991. 
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Microtox  Toxicity  Data  -  5  min  data  from  MEiC 


I 
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Human  Lethal  Chemical  Number 


Lethal  doses  for  50  chemicals  (Wallum,  1998)  are  plotted  in  order  of  increasing  toxicity  as 
concentrations  in  water,  assuming  a  70  kg  person  consuming  5  L  of  water.  Military  Exposure 
Guidelines  (MEGs)  for  water  (5  day,  5  L/day)  are  available  for  15  of  the  50  chemicals 
(USACHPPM,  2001).  Microtox  data  from  Clemedson  et  al.,  1996,  1998. 
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Microtox  Toxicity  Data 
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Military  exposure  guidelines  (MEGs)  for  water  (5  day  exposure,  5  L/day  consxunption)  for  169 
chemicals  (USACHPPM,  2001)  are  plotted  in  order  of  increasing  toxicity.  Corresponding  human 
lethal  concentrations  derived  fix)m  l^al  dose  levels  (Wallum,  1998)  are  available  for  15  of  the  169 
chemicals  and  assume  a  70  kg  person,  5  L  consumption.  Microtox  ^ta  from  Kaiser  and  Palabiica, 
1991. 
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APPENDIX  F 


EVALUATION  DEFINITIONS  AND  METHOD  FOR  THE  ENVIRONMENTAL 
SENTINEL  BIOMONITOR  QUALITIATIVE  SCREENING  MODEL 

Evaluation  Method:  A  two-step  process  will  be  used.  The  first  step  is  to  reduce  the  number 
of  technologies  for  further  consideration  if  they  do  not  meet  a  minimum  requirement.  The 
second  step  is  to  then  assess  and  then  rank  the  remaining  technologies  (or  top  20)  against 
fundamental  criteria. 

Step  1:  Rate  each  ESB  technology  using  the  rating  categories  below. 

Rating  Categories: 

Green  -  means  the  technology  is  promising.  Final  rating  and  ranking 
will  be  completed  in  step  2. 

Orange  -  means  the  technology  may  be  promising  at  some  point  but 
(and)  the  technology  is  currently  immature  and  needs  more  R&D  (e.g., 
the  technology  is  not  to  the  point  in  development  where  testing  can  be 
completed  to  determine  toxicity  sensitivity).  The  technology  will  not 
be  considered  in  step  2  but  is  put  on  a  technical  watch  list. 

Red  -  means  the  technology  is  not  promising  or  not  reasonably 
projected  to  ever  be  promising.  A  technology  earns  a  red  rating  for 
any  of  the  reasons  below  and  is  eliminated  jfrom  further  consideration. 

•  Technology  toxicity  sensitivity  is  inappropriate  or  the 
technology  does  not  provide  a  unique  end-point  (i.e., 
metabolic,  physical  response). 

•  Technology  is  redundant  and  inferior  to  another  technology. 

•  Technology  is  unfeasible/not  viable.  It  does  not  or  is  not 
expected  to  meet  a  minimum  requirement  of  a  user  scenario. 

o  Technology  takes  too  much  time  (hours)  to  produce  an 
end-point. 

o  Technology  requires  pre-culturing  (e.g.,  requires 
actively  taking  out  and  culturing  organism  for  a  long 
period  of  time) 

Step  2:  Evaluate  all  technologies  that  were  rated  green  in  step  1 .  Rate  each  ESB 
technology  against  the  criteria  and  rating  categories  below.  After  each  technology  is 
rated  then  rank  the  technologies  (1-n). 

Rating  Categories: 

Green  -  means  the  ESB  technology  is  known  to  or  is  expected  to 
perform  “well”  against  this  criterion’s  requirements  or  is  “likely”  to 
achieve  this  criterion’s  requirements. 


F-1 


Yellow  -  means  the  ESB  technology  is  known  to  or  is  expected  to 
perform  only  “ok”  against  this  criterion’s  requirements  or  “may” 
achieve  this  criterion’s  requirements. 

Red  -  means  the  ESB  technology  is  known  to  or  is  expected  to 
perform  “poorly”  against  this  criterion’s  requirements  or  is  “not 
likely”  to  achieve  this  criterion’s  requirements. 

Evaluation  Criteria  (see  note  1): 

•  Technology  provides  an  appropriate  toxicity  sensitivity  response 
(SR)  and/or  provides  unique  infonnation  (e.g.,  unique  end-point). 

•  Technology  is  only  minimally  affected  by  interferences  (I). 
Interferences  are  turbidity,  color,  and  ionic  composition  of  the 
media. 

•  Technology  is  reliable/repeatable  (R&R). 

•  Technology  has  an  appropriate  rapidity  of  response  (RoR).  A 
green  rating  is  20  minutes  or  less,  yellow  is  20-60  minutes,  and  red 
is  over  60  minutes. 

Note  1 :  Ranking  decisions  are  based  on  the  ability  of  the  technologies  to  meet 
at  least  one  of  the  user  scenarios.  The  focus  is  primarily  on  appropriate 
toxicity  response  -  at  the  correct  level  of  sensitivity,  in  a  reasonable  period  of 
time,  and  with  the  reproducibility  and  absence  of  interferences  necessary  to 
reach  an  appropriate  evaluation  of  a  test  sample. 
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INFORMATION  ON  SELECTING  TRAINING/TEST  CHEMICALS  AND  LIST  OF 

TRAINING/TEST  CHEMICALS 

Information  on  Selection  of  Training/Test  Chemicals: 

Considering  the  thousands  of  toxic  industrial  and  agro-chemicals,  an  appropriate 
evaluation  of  toxicity  sensor  performance  requires  careful  initial  selection  of  the 
chemicals  to  be  evaluated.  Some  possible  criteria  for  selecting  test  chemicals 
include: 

•  Threat  chemicals.  Several  organizations  (U.S.  Environmental  Protection 
Agency  (EPA),  U.S.  Army  Research  Development  and  Engineering  Center 
(ARDEC))  and  publications  (Garland,  1991;  Hickman,  1999;  Clark  and 
Deininger,  2000)  have  developed  lists  of  threat  chemicals  for  water  supplies. 
Typically,  the  higher  priority  chemicals  are  highly  toxic,  soluble,  persistent  in 
water,  and  available  in  the  region  of  concern.  Chemicals  hi^  on  these  lists 
should  be  given  priority  as  test  chemicals  for  toxicity  sensors. 

•  Mode  of  toxic  action.  Some  chemicals  can  be  grouped  together  because  they 
cause  similar  effects  on  organisms.  Examples  include  organophosphorus  and 
carbamate  insecticides  (which  cause  acetjdcholinesterase  inhibition)  and  many 
neutral  organic  chemicals  such  as  industrial  solvents  (which  cause  narcosis). 
To  the  extent  that  chemicals  can  be  grouped  in  this  way,  it  should  be  possible 
to  select  one  or  two  representative  chemicals  from  the  group  for  testing  to 
establish  whether  a  toxicity  sensor  will  respond  to  chemicals  exhibiting  the 
same  general  mode  of  toxic  action. 

•  Chemical  structural  classes.  Chemicals  with  certain  structural,  functional,  or 
othCT  similarities  are  frequently  grouped  together  for  testing  purposes,  such  as 
heavy  metals,  algal  toxins,  or  habicides.  Since  chemicals  within  these  groups 
may  have  distinctly  different  modes  of  toxic  action,  it  is  best  to  emphasize 
mode  of  toxic  action  categories  in  the  selection  of  test  chemicals.  The  MEIC 
chemical  list  provides  a  broad  range  of  both  modes  of  toxic  action  and 
chemical  structural  classes  (Clemedson  et  al.,  1996, 1998). 

•  Interfering  chemicals  or  water  quality  conditions.  Interfering  chemicals 
include  substances  conmionly  found  in  raw  or  treated  water  that  may  cause 
toxicity  sensor  responses  at  levels  far  below  concentrations  affecting  humans, 
such  as  residual  chlorine,  copper,  or  ammonia.  In  addition,  acceptable  ranges 
of  common  water  quality  conditions,  such  as  temperature,  dissolved  oxygen, 
conductivity,  pH,  turbidity,  and  microbial  concentrations  need  to  be  defined 
for  toxicity  sensors  to  help  minimize  “false  positive”  responses. 

Given  the  broad  range  of  potential  test  substances,  a  compromise  must  be  reached 
between  minimizing  the  niimber  of  chemicals  to  conserve  resources  and  expanding  the 
number  to  better  define  toxicity  sensor  response  characteristics.  One  possible  solution  is  the 
use  of  a  tiered  approach  -  conduct  an  initial  screening  of 
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several  toxicity  sensor  technologies  with  a  few  chemicals,  followed  by  more  thorough 
evaluation  of  technologies  showing  the  most  promise. 

List  of  Training/Test  Chemicals: 

Chemicals  are;  Aldicarb  *,  Ammonia,  Copper  Sulfate,  Mercuric  Chloride  *, 
Methamidophos  *,  Nicotine  *,  Paraquat  Dichloride  *,  Pentachlorophenol  *,  Phenol, 
Sodium  Arsenite  *,  Sodium  Cyanide  *,  and  Toluene. 

Notes:  means  it  is  a  user-selected  chemical 

The  responses  of  each  toxicity  sensor  to  residual  chlorine  and  a  high  conductivity 
freshwater  sample  will  be  determined  as  well. 
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ENVIRONMENTAL  SENTINEL  BIOMONITOR  SYSTEM:  QUANTITATIVE  MODEL 

DESCRIPTION  AND  DETAILS 


Three  separate  multi-criteria  decision-making  models  were  developed.  See 
Figures  I-l  through  /-  3  for  a  visual  representation  of  the  models.  The  wei^ts  for  each  goal 
and  measure  are  also  detailed  in  these  figures.  The  models  (goals,  end-points,  utility  curves) 
are  described  within  each  of  the  ten  measures  below. 

Note:  The  numbers  after  the  measure  name  refer  to  the  26  technical 
requirements  the  user  representatives  developed  for  each  scenario.  Key:  FB  -  Fixed  Base; 
FC  -  Field/Contingenc>^,  SU  -  Small  Unit 

Measure  #:  1 


Measure  Name:  Chemical  Detection  (2) 

GoaKsI:  Performance 

Definition !  The  ability  of  the  technology  to  provide  an  adverse  response  to  a  set  of 
representative  chemicals  at  concentrations  between  the  short-term  MEG  and  the  HLC  for  the 
greatest  number  of  test  chemicals.  For  this  measure,  it  does  not  matter  how  close  the 
technology  provides  an  adverse  response  to  the  MEG. 

What’s  Desired?  It  is  best  to  respond  to  all  (the  best  technologies  detected  6)  chemicals 
between  the  short-term  MEG  and  the  HLC. 

Scale: 

FB/FC/SU  Scenarios: 

100  -  Six  (6)  chemicals  detected  in  the  MEG-HLC  range 

0  -  No  chemicals  detected  in  the  MEG-HLC  range 

Linear  continuous  utility  curve  used 

Scenario  Specifics:  The  performance  scale  and  utility  fimction  for  all  3  scenarios  are  the 
same.  Weight  is  lower  in  FC  and  SU  scenarios  than  the  FB  scenario  due  to  willingness  to 
accept  more  risks  in  field  conditions/situations. 

Notes:  The  scoring  scale  chosen  was  selected  due  to  greater  level  of  discrimination  provided 
by  this  scale.  The  other  scoring  scale,  which  is  a  combination  of  the  number  of  chemicals 
identified  in  the  MEG-HLC  concentration  range  with  a  calculation  of  closeness  of  the  sensor 
response  to  the  MEG,  was  not  selected  because  the  maturity  of  the  sensor  technologies  did 
not  justify  this  level  of  discrimination.  Dilution  of  all  samples  can  be  done  by  calculation  for 


any  technology/sensor  that  detect  consistently  below  the  MEG,  after  which  the  score  would 
be  recalculated  for  that  technology. 


Measure  #:  2 


Measure  Name:  Water  Characteristics  (5,6) 

Goalfs):  Performance 

Definition:  The  ability  to  operate  under  a  number  of  water  quality  conditions  and  in  the 
presence  of  interfering  substances  with  minimal  effect  on  test  outcome.  Performance  scale  is 
based  on  residual  chlorine  as  the  substance  that  provides  the  most  discrimination  between  the 
sensors. 

What’s  Desired?  It  is  better  to  be  able  to  operate  under  a  wide  range  of  water  quality 
conditions  (higher  concentrations  of  residual  chlorine  rather  than  lower  levels)  than  under  a 
more  restricted  range. 

Scale: 

FB/FC/SU  Scenarios: 

100  -  >  5  mg/L  residual  chlorine 

75  -  >  2-5  mg/L  residual  chlorine 

50  - 1-2  mg/L  residual  chlorine 

0  -  <  1  mg/L  residual  chlorine 

Discrete  levels 

Scenario  Specifics:  The  performance  scale  and  utility  function  for  all  3  scenarios  are  the 
same.  Having  to  dechlorinate  or  make  other  water  quality  adjustments  for  the  FB  scenario  is 
not  as  significant  as  having  to  make  adjustments  for  the  SU  scenario. 

Notes: 

No  particular  water  characteristics  are  more  important  than  others. 

Water  characteristics  previously  identified  for  consideration:  quantity  of  total 
dissolved  solids,  turbidity,  clay,  chlorine  residual,  ammonia,  and  metals  [e.g., 
copper],  pH,  temperature,  cations,  chloramines,  and  coagulants. 

Of  these,  most  ESB  technologies  that  require  temp  or  pH  control  include  a 
temp/pH  control  mechanism.  Turbidity  is  not  a  discriminator,  as  no  ESB 
technology  will  operate  properly  in  tuibid  water  (the  water  must  be  filtered). 
Blind  sample  test  data  are  available  for  total  dissolved  solids  and  cations 
(tested  very  high  hardness  water),  residual  chlorine  (up  to  10  mg/L  TRC), 
ammonia,  and  copper.  There  was  only  one  response  below  the  MEG  to 
copper  and  no  responses  below  the  MEG  to  ammonia.  Only  the  S.  meliloti 
system  responded  to  the  high  hardness  water  sample.  Preliminary  data 
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suggest  only  chlorine  has  anv  discriminatory  ability  (at  least  half  the  sensors 
responded  at  concentrations  below  10  mg/L).  Note  that  residual  chlorine  may 
be  removable  by  adding  a  reducing  agent  such  as  sodium  bisulfite,  but  could 
affect  sensor  response  to  other  chemicals  or  change  the  nature  of  the  target 
contaminant. 

The  ability  of  ESB  technologies  to  perform  in  water  with  Humic  acid,  Fulvic  acid, 
and  additional  disinfectants  may  be  an  important  discriminator.  Consider  testing  ESB 
Systems  with  these  interfering  substances  in  the  future. 

Measure  #:  3 

Measure  Name:  Technology  Failure  Rate  (16) 

GoaKsT-  Performance 

Definition:  The  number  of  times  a  technology  fails  to  produce  viable/usable  result  per  a 
certain  number  of  tests.  This  measure  does  not  quantify  failure  because  of  a  human  mistake. 
The  technology  being  evaluated  includes  the  biological  basis  of  the  system  and  all  ancillary 
equipment. 

The  calculation  for  the  failure  rate  is:  [(number  of  tests  attempted)  -  (number  of  tests 
completed)]/  (number  of  tests  attempted)]  x  100% 

What’s  Desired?  It  is  better  to  have  fewer  rather  many  failmes. 

Scale: 

FB  Scenario 

100  -  0%  feilure  rate 

67  -  >  0-5%  failme  rate 

33  ->5-15%  failure  rate 

0  -  >  15%  failme  rate 

Discrete  levels 

FC  Scenario: 

100  -  0%  failure  rate 

80  -  >  0-5%  failure  rate 

30  -  >  5-15%  failure  rate 

0  -  >  15%  failure  rate 

Discrete  levels 

SU  Scenario: 

100  -  0%  failure  rate 

85  -  >  0-5%  feilure  rate 

20  -  >  5-15%  failure  rate 

0  ->  15%  failure  rate 

Discrete  levels 
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Scenario  Specifics:  The  performance  scale  and  utility  flmction  for  all  3  scenarios  are  slightly 
different.  Weight  is  lower  in  FC  and  SU  scenarios  than  the  FB  scenario  due  to  willingness  to 
accept  more  risks  in  field  conditions/situations. 

Notes:  The  best  comparative  data  for  this  metric  is  the  data  provided  by  the  toxicity  sensor 
testers.  They  have  calculated  their  failure  rate  (using  equation  above  in  definition)  for  their 
tests.  In  the  next  phase  of  testing  it’s  important  to  determine  the  four  potential  sensor 
responses  for  each  technology  (True  positives,  True  negatives,  False  positives,  False 
negatives).  More  testing  required  for  project  to  transition  to  next  milestone. 

Measure  Measure  #:  4 


Measure  Name:  Test  Reproducibility  (19) 

GoaKsV.  Performance 

Definition:  The  ability  of  the  ESB  technology  to  produce  the  same  output/answer  given  the 
same  input  and  test  conditions  over  multiple  tests.  This  is  measured  as  median  coefficient  of 
variation;  a  low  coefficient  of  variation  represents  high  reproducibility. 

What’s  Desired?  It  is  best  to  have  high  reproducibility  (a  low  coefficient  of  variation). 

Scale: 

FB/FC/SU  Scenarios: 

100  -  6%  median  coefficient  of  variation 

0  -  54%  median  coefficient  of  variation 

Nonlinear  continuous  curve. 

Scenario  Specifics:  The  performance  scale  and  utility  function  for  all  3  scenarios  are  the 
same.  Weight  is  lower  in  FC  and  SU  scenarios  than  the  FB  scenario  due  to  willingness  to 
accept  more  risks  in  field  conditions/situations. 

Notes:  User  representatives  recommend  giving  those  sensors  with  no  data  available  a  score 
of  zero. 


Measure  Measure  #:  5 


Measure  Name:  Test  Turn  Around  Time  (22, 23) 
GoaKsl:  Operational  Impact 
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Definition:  The  time  required  between  consecutive  tests  (assumes  reusable  technology). 
Includes  operator  set-up  time,  sample  preparation,  sensor  operation  time,  and  any  time 
required  for  the  system  to  reset  before  another  reading. 

What’s  Desired?  It  is  better  to  have  less  rather  than  longer  time  recjuired  between 
consecutive  tests. 

Scale: 

FB/FC/SU  Scenarios 

100  -  5  minutes 

0  -180  minutes 

Nonlinear  continuous  curve. 

Scenario  Specifics:  The  performance  scale  and  utility  fimction  for  all  3  scenarios  are  the 
same. 


Measure  #:  6 


Measure  Name:  Sample  Preparation  and  Skills  Required  (21, 25, 26) 

Goal(s):  Operational  Impact 

Definition:  The  sample  preparation,  performing  test  complexity  level  (e.g.,  measuring 
volume,  adding  reagent),  and  location  of  analysis  (at  sample  site  vs.  in  a  lab).  The  focus  is 
not  on  how  long  it  takes  to  perform  each  task  but  rather  how  complicated  each  task  is. 

What’s  Desired?  It  is  better  for  the  tasks  to  be  simple  (e.g.,  a  soldier  can  perform)  rather  than 
complex  (e.g.,  a  technician  must  perform  at  a  depot). 

Scale: 

FB  Scenario 

100  -  few  steps  of  preparation,  mindmal  skill  level  required  (E4, 9  IS), 

analysis  done  at  sample  site 

67  -  some  steps  of  preparation,  moderate  skill  level  (NCO  and  above, 

water  technician  with  lab  skills  and  lab  capabilities) 

33  -  many  steps  of  preparation,  significant  skill  level  required  (senior  lab 

technician  with  lab  skills  and  lab  capabilities) 

0  -  many  steps  of  preparation,  expert  skill  level  (e.g.;  research  scientist) 

required 

Linear  continuous  curve 
FC  Scenario: 

100  -  few  steps  of  preparation,  no  skill  level  required  (E2,  any  MOS), 

analysis  done  at  sample  site 
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90  -  few  steps  of  preparation,  minimal  skill  level  required  (E4, 9  IS), 

analysis  done  at  sample  site 

50  -  some  steps  of  preparation,  moderate  skill  level  (NCO  and  above, 

water  technician  with  lab  skills  and  lab  capabilities) 

0  -  many  steps  of  preparation,  significant  skill  level  required  (senior  lab 

technician  with  lab  skills  and  lab  capabilities) 

Linear  continuous  cur\'e 

SU  Scenario: 

100  -  few  steps  of  preparation,  no  skill  level  required  (E2,  any  MOS), 

analysis  done  at  sample  site 

70  -  few  steps  of  preparation,  minimal  skill  level  required  (E4, 9 1 S), 

analysis  done  at  sample  site 

0  -  some  steps  of  preparation,  moderate  skill  level  (NCO  and  above, 

water  technician  with  lab  skills  and  lab  capabilities) 

Linear  continuous  curve 

Scenario  Specifics:  The  performance  scale  and  utility  fimction  for  all  3  scenarios  are 
different.  Weight  is  lower  in  FB  scenario  than  FC  and  SU  scenarios  because  of  the 
additional  manpower  and  expertise  available  with  the  FB  scenario. 

E2  is  an  entry  level  of  technical  skill.  E4  would  have  some  lab  training  and  higher  levels  of 
training  and  experience  translates  to  a  senior  lab  technician  who  might  be  an  E6,  E7,  or 
civilian/contractor 


Measure  #:  7 


Measure  Name:  Weight  (8, 11) 

GoaKsl:  Logistics 

Definition:  The  weight  of  the  technology  and  necessary  peripheral  equipment  (e.g., 
maintenance  and  repair  supplies  and  parts,  power  source,  and  a  2-week  supply  of 
consumables). 

What’s  Desired?  It  is  better  for  the  overall  weight  to  be  less  rather  than  more. 

Scale: 

FB/FC/SU  Scenarios: 

100  - 1  lb 

0  -  40  lb 

S-shaped  continuous  curve 
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Scenario  Specifics:  The  performance  scale  is  the  same  for  all  3  scenarios  but  the  utility 
function  is  different  for  each  scenario.  Weight  is  lower  in  FB  scenario  than  the  FC  and  SU 
scenarios  as  there  are  more  options  (e.g.,  carry  in  a  truck  vs.  a  backpack)  and  the  ability  to 
handle  an  ESB  technology  that  wei^  a  lot  (40  lbs  vs.  1  lb). 

Notes:  Size  will  not  be  considered  for  this  evaluation  as  weight  is  considered  a  surrogate  for 
volume/size. 

Measure  #:  8 

Measure  Name:  Shelf  Life  (17) 

Goalfsk  Logistics 

Definition:  The  storage  life  of  tiie  technology  and  consumables  (i.e.,  how  long  will  they  be 
usable  [includes  shipping  time]  under  optimal  conditions  for  that  technology). 

What’s  Desired?  It  is  better  when  the  technology  and  its  consumables  have  a  long  shelf  life. 

Scale: 

FB  Scenarios 

100  -  greater  than  12  months 

0  -  2  weeks  (1/2  month)  of  less 

Non-linear  continuous  curve 

FC  Scenario: 

100  -  greater  than  12  months 

75  -  6-12  months 

40  -  3-6  months 

0  -  3  months  or  less 

Discrete  levels 

SU  Scenario: 

100  -  greater  than  12  months 

75  -  6-12  months 

30  -3-6  months 

0  -  3  months  or  less 

Discrete  levels 
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Scenario  Specifics:  The  performance  scale  and  utility  function  for  all  3  scenarios  are 
different.  Weight  is  lower  in  FB  scenario  than  the  FC  and  SU  scenarios  as  there  are  more 
options  and  the  ability  to  handle  an  ESB  technology  with  a  short  shelf  life. 

Notes:  Expert  input  estimates  shelf  life  for  ESB  technology  components  to  range  from  2 
weeks  to  2  years. 


Measure  #:  9 


Measure  Name:  Storage  Conditions  Required  (17) 

Goal(s):  Logistics 

Definition:  The  conditions  required  to  maintain  the  technology  in  operating  condition  for  the 
maximum  possible  shelf  life. 

What’s  Desired?  The  least  special  or  demanding  storage  conditions. 

Scale: 

FB  Scenario: 

100  -  Easy,  ambient  conditions  (includes  extremes  up  to  71  degrees  C) 

50  -  Average,  controlled  room  temperature 

30  -  Demanding,  refiigeration  required 

0  -  Extremely  demanding,  very  low  temp  or  narrow  temp  range  required 

Discrete  Levels 
FC  Scenario: 

100  -  Easy,  ambient  conditions  (includes  extremes  up  to  71  degrees  C) 

50  -  Average,  controlled  room  temperature 

0  -  Demanding,  refiigeration  required 

Discrete  Levels 

SU  Scenario: 

100  -  Easy,  ambient  conditions  (includes  extremes  up  to  71  degrees  C) 

0  -  Controlled  room  temperature 

Discrete  Levels 

Scenario  Specifics:  The  performance  scale  and  utility  function  for  all  3  scenarios  are 
different.  Weight  is  lower  in  FB  scenario  than  the  FC  and  SU  scenarios  as  there  are  more 
options  and  the  ability  to  handle  an  ESB  system  that  requires  demanding  storage  condition. 
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Measure  #:  10 


Measure  Name:  Biological  Component  Maintenance  (17) 

Goalfs'):  Logistics 

Definition:  Time  required,  complexity  of  system,  and  material  and  logistics  required  to 
maintain  biological  component  of  the  test  system. 

What’s  Desired?  Requires  no  biological  culturing.  Time  required  to  maintain  biological 
component  is  minimal,  system  to  mamtain  is  simple,  and  requirements  to  maintain  organism 
culture  is  minimal. 

Scale: 

FB  Scenario 

100  -  requires  no  biological  culturing 

60  -  requires  some  maintenance  but  no  culture  medium  exchange 

0  -  requires  culture  medium  exchange  or  C02  for  culture  medium 

Discrete  levels 

FC  Scenario: 

100  -  requires  no  biological  culturing 

40  -  requires  some  maintenance  but  no  culture  medium  exchange 

0  -  requires  culture  medium  exchange  or  C02  for  culture  mediiim 

Discrete  levels 

SU  Scenario: 

100  -  requires  no  biological  culturing 

0  -  requires  some  maintenance  but  no  culture  medimn  exchange 

Discrete  levels 

Scenario  Specifics:  The  performance  scale  and  utility  function  for  all  3  scenarios  are 
different.  Weight  is  lower  in  FB  scenario  than  the  FC  and  SU  scenarios  as  there  are  more 
options  and  the  ability  to  handle  an  ESB  system  that  requires  biological  component 
maintenance. 
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Figure  I-  1;  Fixed  Base  Scenario  Model  with  Weights 
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Figure  1-3;  Small  Unit  Scenario  Model  with  Weights 
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of  the  limited  time  the  technical  experts  had  to  complete  this  assignment,  tiie  programmatic  input/comments  are  very  general. _ 

Note  2:  Costs  don't  include  initial  technology  development  costs  but  rather  only  costs  to  initially  buy  final  technology  product  and  maintain  it  to 

include  supplies  to  operate  it. _ _ _ _ _ 

Note  3:  Maturity  level  of  technology  is  considered  in  evaluation. _ _ _ 
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ENVIRONMENTAL  SENTINEL  BIOMONITOR  SYSTEM:  TECHNOLOGY 
STRENGTHS  AND  WEAKNESSES  ANALYSIS 

The  purpose  of  the  technology  analysis  is  to  highli^t  areas  where  particular 
technologies  stand  out,  eitiier  positively  or  negatively.  This  appendix  contains  narrative 
summaries  {Table  K-1)  of  the  relative  strengths  and  weaknesses  for  each  technology,  as  well 
as  charts  {Figures  K-1  -  K-1 2)  that  show  how  each  technology  performed  relative  to  the  ten 
measures. 


For  the  narrative  summaries  “average/below  average”  descriptors  are  used. 
These  were  subjectively  determined  by  the  DAT  and  are  based  on  each  ESB  technology’s 
score  relative  to  the  other  ESB  technologies. 

The  twelve  ESB  technologies  listed  below  were  evaluated  by  die  DAT  against 
how  they  scored  relative  to  the  Fixed  Base  Scenario  model.  They  are  presented  in  the  order 
that  they  scored  against  the  Fixed  Base  Scenario  model  (i.e.,  number  in  parenthesis  is  score 
[e.g.,  82  for  Microtox]). 

Table  K-1:  Technology  Strengths  and  Weaknesses  Analysis  Narrative  Siraunaries 


1. 

Microtox  (82)  scored  above  average  on  seven  measures  {Chemical  Detection, 

Technology  Failure  Rate,  Test  Turn  Around  Time,  Preparation  and  Skills  Required, 

Shelf  Life,  and  Biological  Component  Maintenance)  and  below  average  on  Water 
Characteristics,  Weight,  and  Storage  Conditions  Required  measures.  Microtox 
detected  six  chemicals,  including  one  chemical  that  the  other  three  top-ranked 
technologies  did  not  detect  -  cyanide. 

2. 

Hepatocyte  Low  Density  Lipoprotein  Uptake  (72)  scored  above  average  on  four 
measures  {Chemical  Detection,  Test  Reproducibility,  Water  Characteristics,  and 
Biological  Component  Maintenance)  and  below  average  on  all  other  measures. 
Hepatocyte  Low  Density  Lipoprotein  Uptake  detects  six  chemicals  and  one  chemical 
the  other  three  technologies  in  the  top  four  ranked  technologies  combined  don’t  detect  - 
paraquat. 

3. 

Electric  Cell-Substrate  Impedance  Sensing  (67)  scored  above  average  on  two  measures 
(Chemical  Detection  and  Water  Characteristics)  and  below  average  on  Technology 
Failure  Rate,  Test  Turn  Around  Time,  Sample  Preparation  and  Skills  Required,  Weight, 
Shelf  Life,  Storage  Conditions  Required,  and  Biological  Component  Maintenance. 
Electric  Cell-Substrate  Impedance  Sensing  detects  one  chemical,  ammonia,  that  the  top 
four  ranked  technologies  combined  do  not  detect. 

Mitoscan  Electron  Transfer  (65)  scored  above  average  on  six  measures  {Water 
Characteristics,  Technology  Failure  Rate,  Test  Reproducibility,  Weight,  Shelf  Life,  and 
Biological  Component  Maintenance)  and  below  average  on  Chemical  Detection, 

Sample  Preparation  and  Skills  Required,  and  Storage  Conditions  Required.  Mitoscan 
Electron  Transfer  does  not  detect  any  chemicals  that  the  top  four  ranked  technologies 
combined  do  not  detect. 

5. 

Eclox  (55)  scored  above  average  on  seven  measures  (Technology  Failure  Rate,  Test 
Reproducibility,  Test  Turn  Around  Time,  Sample  Preparation  and  Skills  Required,  Shelf 
Life,  Storage  Conditions  Required,  and  Biological  Component  Maintenance)  and  below 
average  for  Chemical  Detection  and  Water  Characteristics  measures.  Eclox  only 
detects  one  chemical  -  copper.  The  top  four  scored  technologies  also  detect  copper. 

6. 

Neuronal  Microelectrode  Array  (52)  scored  above  average  for  the  Test  Reproducibility 
measure  and  below  average  on  six  measmes  (Technology  Failure  Rate,  Test  Turn 

Around  Time,  Preparation  and  Skills  Required,  Weight,  Shelf  Life,  and  Storage 
Conditions  Required).  But  the  Neuronal  Microelectrode  Array  detects  two  chemicals 
the  top  four  rated  technologies  do  not  detect  (aldicaib  and  methamidophos  - 
organophosphates). 

1 

SOS  Cytosensor  (51)  scored  above  average  in  three  measures  (Chemical  Detection, 

Water  Characteristics,  and  Storage  Conditions  Required)  and  below  average  on  Test 
Reproducibility,  Test  Turn  Around  Time,  Preparation  and  Skills  Required,  and  Shelf 

Life  measures.  SOS  Cytosensor  detects  five  chemicals  although  it  does  not  detect  any 
chemicals  that  the  top  four  rated  technologies  do  not  detect. 

8. 

Toxi-Chromotest  (49)  scored  above  average  on  three  measures  (Water  Characteristics, 
Technology  Failure  Rate,  and  Shelf  Life)  and  below  average  on  Chemical  Detection, 

Test  Turn  Around  Time,  and  Storage  Conditions  Required.  Toxi-Chromotest  only 
detects  two  chemicals  and  these  chemicals  are  not  different  fix)m  the  chemicals  detected 
by  the  hipest  four  scored  technologies. 

9. 

ToxScreen  Organic  (tied  with  ToxScreen  Metal,  42)  scored  above  average  on  three 
measures  (Preparation  and  Skills  Required,  Weight,  and  Shelf  Life)  and  below  average 
on  Water  Characteristics,  Technology  Failure  Rate,  Test  Reproducibility,  and  Test 

Turn  Around  Time.  The  three  chemicals  detected  are  not  different  fi'om  the  chemicals 
detected  by  the  highest  four  scored  technologies. 

10. 

ToxScreen  Metal  (tied  with  ToxScreen  Organic,  42)  scored  above  average  on  three 
measures  (Preparation  and  Skills  Required,  Weight,  and  Shelf  Life)  and  below  average 
on  Technology  Failure  Rate,  Test  Reproducibility,  and  Test  Turn  Around  Time.  The 
three  chemicals  detected  are  not  different  fi-om  the  chemicals  detected  by  the  hipest 
four  scored  technologies. 

11. 

Sinorhizobium  meliloti  Assay  (40)  scored  above  average  on  five  measures  (Water 
Characteristics,  Test  Turn  Around  Time,  Preparation  and  Skills  Required,  Weight  and 
Storage  Conditions  Required)  and  below  average  on  Test  Reproducibility,  Shelf  Life, 
and  Biological  Component  Maintenance  measures.  It  detected  three  chemicals,  none 
that  the  highest  four  scored  technologies  do  not  also  detect. 

12. 

Mitoscan  Reverse  Electron  Transfer  (33)  scored  above  average  on  four  measures 
(Technology  Failure  Rate,  Weight,  Shelf  Life,  and  Biological  Component  Maintenance) 
and  below  average  on  Chemical  Detection,  Preparation  and  Skills  Required,  Weight, 
and  Storage  Conditions.  It  detected  one  chemical,  none  that  the  highest  four  scored 
technologies  do  not  also  detect. 

Each  chart  below,  Figures  K-1  -  K-12,  represent  a  different  ESB  technology. 
They  depict  the  strengths  and  weaknesses  for  each  of  the  17  ESB  technologies  relative  to  the 
ten  measures  in  the  Fixed  Base  Scenario  model.  The  height  of  the  bars  indicates  a 
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technology’s  relative  score  for  each  measure,  while  the  width  indicates  the  relative  wei^t  of 
the  measure.  For  Microtox,  for  example,  the  chart  shows  it  scored  very  hi^  against  four 
measures,  which  are  Chemical  Detection  (highest  weighted  measure  also).  Shelf  Life, 
Biological  Component  Maintenance,  and  Technology  Failure  Rate.  The  measures  where 
Microtox  scored  low  were  generally  weighted  low,  for  example  the  Water  Characteristics 
measure  (H20  Char,  5th  bar  fix)m  the  left).  The  charts  are  provided  in  the  order  of  overall 
technology  ranking  results  fi’om  best  to  worst  (based  on  Fixed  Base  Scenario  model  results). 
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Figure  K-1;  Microtox  Strengths  and  Weaknesses  Chart 
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Figure  K-2;  Hepatocyte  Low  Density  Lipoprotein  Uptake  Strengths  and  Weaknesses  Chart 
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Figure  K-3:  Electric  Cell-Substrate  Impedance  Sensing  Strengths  and  Weaknesses  Chart 
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Figure  K-4:  Mitoscan  Electron  Transfer  Strengths  and  Weaknesses  Chart 
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Figure  K-5:  Eclox  Strengths  and  Weaknesses  Chart 
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Figure  K-6:  Neuronal  Microelectrode  Array  Strengths  and  Weaknesses  Chart 
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Figure  K-7:  SOS  Cytosensor  Strengths  and  Weaknesses  Chart 
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Figure  K-8:  Toxi-Chromotest  Strengths  and  Weaknesses  Chart 
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Figure  K-9:  ToxScreen  Organic  Strengths  and  Weaknesses  Chart 
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Figure  K-10:  ToxScreen  Metal  Strengths  and  Weaknesses  Chart 
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Figure  K-11:  Sinorhizobium  meliloti  Assay  Strengths  and  Weaknesses  Chart 
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Figure  K-12:  Mitoscan  Reverse  Electron  Transfer  Strengths  and  Weaknesses  Chart 
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ENVIRONMENTAL  SENTINEL  BIOMONITOR  SYSTEM:  QUANTITATIVE  MODEL 

MEASURES  ANALYSIS 

The  measures  analysis  summarizes  the  assessment  results  in  terms  of 
individual  measmes.  Because  the  measures  represent  user  needs,  this  analysis  helps  identify 
areas  of  shortfall,  or  potential  technical  challenges  (e.g.,  if  the  ESB  technologies  generally 
score  low  against  a  measure).  Conversely,  the  analysis  also  identifies  areas  of  minimal 
concern  (e.g.,  the  technologies  generally  score  high  and/or  the  measme  is  low-weighted). 

The  range  of  scores  for  each  measure  was  examined  to  determine  overall  ESB 
technology  performance  relative  to  each  measure,  which  provided  the  basis  for  assignments 
of  a  subjective  assessment  rating  by  the  DAT.  Comments  and  rationale  for  each  measure 
were  also  provided.  Note  that  all  measures  were  evaluated  as  either  a  green  or  yellow  rating. 
Table  L-1  summarizes  the  results  from  the  measure  analysis  along  with  defining  subjective 
ratings. 

Table  L-1:  Measure  Analysis 


Measures 
(Weight  -  see 
Note  1) 

Rating 

(see 
Note  2) 

Comments 

Chemical 
Detection  (0.49) 

1 

-  This  measure  is  wei^ted  the  highest,  almost  half  of  the  models 
weight. 

-  The  utility  function  is  a  linear  continuous  curve.  For  each  chemical 
detected  the  technology  earns  16.7%  of  the  possible  score.  For 
exanq>le,  detecting  four  chemicals  would  earn  a  score  of  66.8  (4  x  16.7) 
out  of  100. 

-  Seven  technologies  detected  three  chemicals  or  fewer  and  thus  earned 
half  or  less  of  the  possible  maximum  score. 

Water 

Characteristics 

(0.04) 

Green 

-  This  measure  is  weighted  very  low  and  thus  has  a  small  overall 
inq)act  on  the  models  score. 

-  The  utility  function  is  four  discrete  levels.  In  this  function  it  is 
important  to  be  able  to  operate  in  water  with  at  least  1  mg/L  of  residual 
chlorine  because  at  least  half  of  the  total  possible  score  is  earned  by 
doing  so. 

-  Six  technologies  scored  at  the  highest  level  for  the  measure  (i.e.,  they 
could  operate  in  water  with  over  5  mg/L  of  residual  chlorine),  three 
scored  at  the  50%  point  (i.e.,  can  handle  residual  chlorine  between  1-2 
mg/L  only),  and  three  scored  at  the  lowest  level  (i.e.,  they  could  operate 
in  water  with  less  than  1  mg/L  of  residual  chlorine). 

-  Dechlorinating  the  water  before  using  the  ESB  technology  can 
mitigate  issues  with  residual  chlorine. 

Technology 
Failure  Rate 
(0.03) 

Green 

-  This  measure  is  weighted  very  low  and  thus  has  a  small  overall 
impact  on  the  models  score. 

-  The  utility  function  is  four  discrete  levels.  The  possible  range  of 
failure  rates  range  fiom  0-15+%  with  any  rates  between  5-15%  earning 
a  relatively  low  33  out  of  100  score. 

-  Results  for  this  measure  were  determined  from  controlled 
experiments. 

-  Five  technologies  scored  at  the  highest  level  (i.e.,  0%  failure  rate). 
Neuronal  Microelectrode  Array  is  the  only  technology  that  scored  a 
zero  for  this  measure  (i.e.,  more  than  15%  failure  rate).  Technical 
Experts  felt  this  failure  rate  could  be  significantly  reduced  by  fixing 
know  shipping  problems. 

Test 

Reproducibility 

(0.23) 

Yellow 

-  This  measure  is  weighted  the  second  highest,  almost  a  quarter  of  the 
models  weight. 

-  The  utility  function  is  a  non-linear  continuous  curve.  Having  a 
median  coefficient  of  variation  of  20%  earns  about  30  out  of  100  score. 
Having  a  median  coefficient  of  variation  of  12%  earns  about  a  70  out  of 
100  score.  There  is  a  shaip  increase  in  score  for  earning  a  median 
coefficient  of  variation  of  greater  than  20%. 

-  Two  technologies  scored  very  well  (Eclox  and  Mitoscan  Electron 
Transfer),  four  scored  very  low  (SOS  Cytosensor,  ToxScreen  Organic, 
ToxSceen  Metal,  and  Sinorhizobium  meliloti  Assay),  and  the  rest 
scored  in  the  middle. 

Test  Turn 

Around  Time 
(0.05) 

Yellow 

-  This  measure  is  weighted  low  and  thus  has  a  small  overall  impact  on 
the  models  score. 

-  The  utility  function  is  a  non-linear  continuous  curve.  It  is  much  more 
important  to  score  closer  to  5  minutes  than  180  minutes.  A  turn  around 
time  of  20  minutes  earns  a  score  of  75  out  of  100  score  while  40 
minutes  earns  40  out  of  100  score  and  140  minutes  earns  less  than  5  out 
of  100  score. 

-  Nine  technologies  scored  less  than  25  out  of  100. 

Preparation  and 
Skills  Required 
(0.04) 

Green 

-  This  measure  is  weighted  very  low  and  thus  has  a  small  overall 
impact  on  the  models  score. 

-  The  utility  function  is  four  discrete  levels.  The  lowest  level  scores  0 
out  of  100.  This  lowest  level  requires  many  sample  preparation  steps 
and/or  expert  skills  to  prepare  the  sample.  The  other  three  levels  are 
each  worth  1/3, 2/3,  and  100%  of  measures  score. 

-  Two  technologies  scored  a  0,  Neuronal  Microelectrode  Array  and 

SOS  Cytosensor,  which  means  the  skills  of  a  research  scientist  with  a 
lab  are  required. 

Weight  {Q.Q\) 

Green 

-  This  measure  is  tied  with  the  Storage  Conditions  Required  measure  as 
the  lowest  weighted  measures  and  thus  has  a  small  overall  impact  on 
the  models  score. 

-  The  utility  function  is  a  non-linear  continuous  curve  (backwards  s- 
shape).  It  is  much  more  important  to  score  closer  to  1  lb  vs.  40  lbs. 

For  example,  a  technology  weighting  10  lbs  earns  a  score  of  85  out  of 
100,  20  lbs  earns  a  50,  and  30  lbs  earns  a  10. 

-  Six  technologies  scored  above  70,  two  score  between  40-50,  and  three 
scored  below  5  out  of  100. 
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Shelf  Life  {Om) 

Yellow 

-  This  measure  is  weighted  low  and  thus  has  a  sm^  overall  impact  on 
the  models  score. 

The  utility  function  is  a  non-linear  continuous  curve.  Although,  it  is 
important  to  have  a  shelf  life  for  12-months  it  is  most  important  to  have 
a  shelf  life  of  at  least  3-months  as  this  earns  a  score  of  50  out  of  100 
and  6-months  earns  an  80. 

-  Five  technologies  have  a  shelf  life  of  one  month  or  less.  All  other 
technologies  have  a  shelf  life  of  longer  than  six  months. 

Storage 
Conditions 
Required  (0.01) 

Yellow 

-  This  measure  is  tied  with  the  Weight  measure  as  the  lowest  weighted 
measures  and  thus  has  a  small  overall  impact  on  the  models  score. 

-  The  utility  function  is  four  discrete  levels.  The  average  level,  which 
is  defined  further  as  controlled  room  temperature,  earns  50  out  of  100. 
The  two  levels  below  average  earn  a  30  and  0  out  of  100.  Thus  any 
technology  requiring  any  temperature  control  scores  low  for  this 
measure. 

-  Seven  technologies  scored  at  the  lowest  level.  Only  one  technology, 
Eclox,  scored  at  the  highest  level  and  one  scored  at  the  average  level  - 
SOS  Cytosensor. 

Biological 

Component 

Maintenance 

(0.03) 

Green 

-  This  measure  is  weighted  very  low  and  thus  has  a  small  overall 
impact  on  the  models  score. 

-  The  utility  function  is  three  discrete  levels.  If  a  technology  requires 
any  culture  medium  exchange  it  scores  0  out  of  100. 

-  Nine  technologies  scored  at  the  highest  level  but  two  at  the  lowest 
level  meaning  the  technology  requires  culture  medium  exchange  or 

C02  for  culture  medium. 

Note  1 :  Weights  (e,g.,  0.49  for  Chemical  Detection  measure)  is  for  the  Fixed  Base  Scenario 
model. 

Note  2:  Rating  Key: 

Green:  Area  of  low  concern;  indicating  user  needs  should  be  met,  or  the  measure  is 
low  weight  and  is  unlikely  to  cause  significant  impact. 

Yellow:  Area  of  moderate  concern;  indicating  user  needs  may  not  be  met. 

Red:  Area  of  hi^  concern;  indicating  user  needs  are  not  readily  met. 

Note  3:  Became  the  maximum  number  of  chemicals  detected  by  any  technology  is  six  this  is 
the  100  score.  It  is  a  goal  to  detect  all  twelve  chemicals,  though.  In  order  to  detect  more 
than  six  chemicals  an  ESB  technology  system  is  needed  (i.e.,  two  or  more  technologies 
working  together). 
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ENVIRONMENTAL  SENTINEL  BIOMONITOR  SYSTEM:  MEASURE  SENSITIVITY 

ANALYSIS  CHARTS 

Sensitivity  analysis  was  also  perfonned  to  determine  whdher  changes  in 
measures  weights  would  affect  die  results.  This  analysis  was  coiniucted  on  all  measures. 
LDW  was  used  to  generate  soisitivity  graphs  for  each  measure.  The  sensitivity  gn^hs 
contain  the  ovoall  score  on  the  y-axis  worst  to  best)  and  the  percent  of  &e  weight 
given  to  the  measure  being  evaluated  on  die  x-axis  (fiom  0  to  100).  There  is  a  line  segment 
for  each  ESB  technology  on  the  grqih,  which  extends  from  die  x-axis  at  various  angles  to  the 
position  that  corresponck  with  100%  of  the  weight  being  assigned  to  that  measure.  A  vertical 
line  on  the  sensitivity  graph  identifies  the  curroit  wei^t  assigned  to  the  measure.  The  order 
in  which  the  vertical  wei^t  line  intersects  die  alternatives’  segments  is  die  overall  rank  order 
produced  by  the  model’s  weighting  scheme. 

By  visually  noting  the  changes  in  rank  order  which  would  occur  if  the  vertical 
weight  line  was  moved  to  the  right  or  left  (measure  weight  increased  or  decreased),  an 
assessment  can  be  made  as  to  the  soisitivity  of  the  wei^t  of  that  measure.  If  a  slight 
movement  of  the  wei^t  line  causes  multiple  alternative  lines  to  cross  (and  therefore  die  rank 
order  to  have  multiple  changes),  then  that  measure  would  be  considered  very  sensitive  to 
weight  changes.  I^  on  the  other  hand,  no  ranking  changes  occur  when  the  line  is  moved  a 
greater  distance,  then  it  can  be  determined  that  the  weight  of  that  measure  does  not  have  a 
notable  effect  on  die  outcomes  of  the  analysis. 

For  the  most  part,  the  technology  rankings  are  insensitive  to  reasonable  (:t 
15%  changes)  measure  weight  changes.  This  is  especially  true  for  die  top  three  scoring 
technologies  (Microtox,  Hepatocyte  Low  Density  lipoprotein  Uptake,  and  Electric  Cell- 
Siibstrate  Impedance  Sensing).  Diere  are  two  instances  were  reasonable  weight  changes 
have  an  impact: 


•  The  first  instance  is  when  the  Chemical  Detection  measure  weight  is 
inouased  or  reduced  15%  (considoed  maximum  reasonable  change;  see 
Figure  M-1).  By  decreasing  the  wei^t  of  die  Chemical  Detection 
measure  15%  Mitoscan  Electron  Transfer  and  Eclox  change  positions  in 
die  ranking  (3*^  and  4^  positions).  Toxi-Chromotest  increases  from  8^ 
highest  ranked  to  6^  hipest  ranked.  By  inoeasing  die  wei^t  of  die 
Chemical  Detection  measure  15%,  the  impact  on  die  ranking  is  a  litde  less 
as  only  SOS  Cytosensor  increases  fiom  7™  highest  ranked  to  5^  highest 
ranked  and  exchanges  ranking  with  Eclox. 

•  The  second  instance  is  whoi  die  wei^t  of  the  next  highest  weighted 
measure.  Test  Reproducibility,  decreases  15%  (see  Figure  M-4).  When 
this  h^ipens,  SOS  Cytosensor  increases  fiom  seventh  hipest  ranked  to 
fifth  highest  ranked.  Eclox  and  Neuronal  Microelectrode  Array  each  drop 
one  position  in  ranking.  When  the  measure  is  increased  15%  there  is  only 
a  sli^t  impact  -  Electric  Cell-Substrate  Impedance  Sensing  and  Mitoscan 


•  Electron  Transfer  would  rank  the  same  and  SOS  Cytosensor  and  Toxi- 
Chromotest  would  rank  the  same. 

Sensitivity  analysis  was  conducted  for  all  ten-evaluation  measures  for  all  three 
models/scenarios.  Sensitivity  graphs  for  the  measures  for  the  Fixed  Base  Scenario  are 
provided  in  Figures  M-1  -  M-12. 
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Figure  M-1 :  Chemical  Detection  Measure  Sensitivity  Chart 
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Figure  M-2:  Water  Characteristics  Measure  Sensitivity  Chart 


APPENDIX  M 


M-2 


t 


-  Micro tox 

-  Hepatocyte  LDL 

-  ECIS 

-  Mitoscan  ETR 

-  Eclox 

-  Neur  Microel  Array 

-  SOSCyto 

-  Toxi-Chrom  Test 

-  ToxScreen  Org 

-  ToxScreen  Metal 

-  S  meliloti 

-  Mitoscan  RET 


Figure  M-3;  Technology  Failure  Rate  Measure  Sensitivity  Chart 


i 

1 


Best 

1 

1 

^  _  ^ 

Utility  ' 

Worst 

- - - 

-1  ) 

1  1  1  j  1  1 

1 

1  1 

0 

1  1  1  (  I  1  I  ^ 

100 

Percent  of  Weight  on  Test  Reproducibility  Measure 
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Figure  M-4:  Test  Reproducibility  Measure  Sensitivity  Chart 
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Figure  M-5:  Test  Turn  Around  Time  Measure  Sensitivity  Chart 
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Figure  M-6:  Sample  Preparation  and  Skills  Required  Measure  Sensitivity  Chart 
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Figure  M-7:  Weight  Measure  Sensitivity  Gr^h 
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Figure  M-8:  Shelf  Life  Measure  Sensitivity  Chart 
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Figure  M-9:  Storage  Conditions  Required  Measure  Sensitivity  Chart 
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Figure  M-10:  Biological  Component  Maintenance  Measure  Sensitivity  Chart 
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